Emergency Department (ED) crowding is a major public health challenge since it can seriously impact patient outcomes; and accurate prediction of patient flow in EDs is essential for improving operational efficiency and quality of care. We present a deep learning framework to predict patient flow rates in EDs, namely the rates of arrival, treatment, and discharge for patients across all triage levels. Our model detects short-term and long-term temporal dependencies within the time-series data of a given patient-flow variable, as well as dependencies between time-series data of different patient-flow variables. We implement this framework as a convolutional neural network, which we call PatientFlowNet. Our proposed model learns simultaneously from multiple flow variables over a long temporal window and predicts future values of arrival, treatment, and discharge rates in the ED. We benchmark our model against state-of-the-art methods on data from EDs in three different hospitals. Results show that PatientFlowNet achieves superior prediction accuracy, compared to the baseline methods, and yields a mean absolute error that is 4.8% lower than the leading baseline. Furthermore, we provide a visual and interpretable representation of the learned dependencies by our model, between patient-flow variables in EDs.