You have a file with data you want to process with Pandas, and you want to make sure you won’t run out of memory. How do you estimate memory usage given the file size? At times you may see estimates like these: “Have 5 to 10 times as much RAM as the size of your dataset”, or “several times the size of your dataset”, or 2×-3× the size of the dataset. All of these estimates can both under- and over-estimate […]

Most modern web applications are powered by a REST API under the hood. That way, developers can separate JavaScript front-end code from the back-end logic that a web framework like Flask provides. Following this step-by-step project, you’ll create an interactive single-page application with HTML, CSS, and JavaScript. The foundation is an existing Flask project with a REST API and a connected SQLite database, which you’ll grab in just a moment. In this tutorial, you’ll learn how to: Navigate a full-stack […]

One of the most common questions you might have when entering the world of pandas is how to iterate over rows in a pandas DataFrame. If you’ve gotten comfortable using loops in core Python, then this is a perfectly natural question to ask. While iterating over rows is relatively straightforward with .itertuples() or .iterrows(), that doesn’t necessarily mean iteration is the best way to work with DataFrames. In fact, while iteration may be a quick way to make progress, relying […]

Libraries like NumPy and Pandas let you switch data types, which allows you to reduce memory usage. Switching from numpy.float64 (“double-precision” or 64-bit floats) to numpy.float32 (“single-precision” or 32-bit floats) cuts memory usage in half. But it does so at a cost: float32 can only store a much smaller range of numbers, with less precision. So if you want to save memory, how do you use float32 without distorting your results? Let’s find out! In particular, we will: Explore some […]

The Python standard shell, or REPL (Read-Eval-Print Loop), allows you to run Python code interactively while working on a project or learning the language. This tool is available in every Python installation, so you can use it at any moment. As a Python developer, you’ll spend a considerable part of your coding time in a REPL session because this tool allows you to test new ideas, explore and experiment with new tools and libraries, refactor and debug your code, and […]

Linear algebra is an important topic across a variety of subjects. It allows you to solve problems related to vectors, matrices, and linear equations. In Python, most of the routines related to this subject are implemented in scipy.linalg, which offers very fast linear algebra capabilities. In particular, linear models play an important role in a variety of real-world problems, and scipy.linalg provides tools to compute them in an efficient way. In this tutorial, you’ll learn how to: Study linear systems […]

If you need to speed up Python, Cython is a very useful tool. It lets you seamlessly merge Python syntax with calls into C or C++ code, making it easy to write high-performance extensions with rich Python interfaces. That being said, Cython is not the best tool in all circumstances. So in this article I’ll go over some of the limitations and problems with Cython, and suggest some alternatives. A quick overview of Cython In case you’re not familiar with […]