# Question

How do I convert a pandas dataframe into a NumPy array?

DataFrame:

``````import numpy as np
import pandas as pd
``````index = [1, 2, 3, 4, 5, 6, 7]
a = [np.nan, np.nan, np.nan, 0.1, 0.1, 0.1, 0.1]
b = [0.2, np.nan, 0.2, 0.2, 0.2, np.nan, np.nan]
c = [np.nan, 0.5, 0.5, np.nan, 0.5, 0.5, np.nan]
df = pd.DataFrame({‘A’: a, ‘B’: b, ‘C’: c}, index=index)
df = df.rename_axis(‘ID’)
``````

gives

``````label   A    B    C
ID
1   NaN  0.2  NaN
2   NaN  NaN  0.5
3   NaN  0.2  0.5
4   0.1  0.2  NaN
5   0.1  0.2  0.5
6   0.1  NaN  0.5
7   0.1  NaN  NaN
``````

I would like to convert this to a NumPy array, like so:

``````array([[ nan,  0.2,  nan],
[ nan,  nan,  0.5],
[ nan,  0.2,  0.5],
[ 0.1,  0.2,  nan],
[ 0.1,  0.2,  0.5],
[ 0.1,  nan,  0.5],
[ 0.1,  nan,  nan]])
``````

Also, is it possible to preserve the dtypes, like this?

``````array([[ 1, nan,  0.2,  nan],
[ 2, nan,  nan,  0.5],
[ 3, nan,  0.2,  0.5],
[ 4, 0.1,  0.2,  nan],
[ 5, 0.1,  0.2,  0.5],
[ 6, 0.1,  nan,  0.5],
[ 7, 0.1,  nan,  nan]],
dtype=[('ID', '<i4'), ('A', '<f8'), ('B', '<f8'), ('B', '<f8')])
``````

# Use `df.to_numpy()`

It's better than `df.values`, here's why.*

It's time to deprecate your usage of `values` and `as_matrix()`.

pandas v0.24.0 introduced two new methods for obtaining NumPy arrays from pandas objects:

1. `to_numpy()`, which is defined on `Index`, `Series`, and `DataFrame` objects, and
2. `array`, which is defined on `Index` and `Series` objects only.

If you visit the v0.24 docs for `.values`, you will see a big red warning that says:

### Warning: We recommend using `DataFrame.to_numpy()` instead.

* - `to_numpy()` is my recommended method for any production code that needs to run reliably for many versions into the future. However if you're just making a scratchpad in jupyter or the terminal, using `.values` to save a few milliseconds of typing is a permissable exception. You can always add the fit n finish later.

## Towards Better Consistency: `to_numpy()`

In the spirit of better consistency throughout the API, a new method `to_numpy` has been introduced to extract the underlying NumPy array from DataFrames.

``````# Setup
df = pd.DataFrame(data={'A': [1, 2, 3], 'B': [4, 5, 6], 'C': [7, 8, 9]},
index=['a', 'b', 'c'])
Convert the entire DataFrame
df.to_numpy()
array([[1, 4, 7],
[2, 5, 8],
[3, 6, 9]])
Convert specific columns
df[[‘A’, ‘C’]].to_numpy()
array([[1, 7],
[2, 8],
[3, 9]])
``````

As mentioned above, this method is also defined on `Index` and `Series` objects (see here).

``````df.index.to_numpy()
# array(['a', 'b', 'c'], dtype=object)
df[‘A’].to_numpy()
array([1, 2, 3])
``````

By default, a view is returned, so any modifications made will affect the original.

``````v = df.to_numpy()
v[0, 0] = -1
``````df
A  B  C
a -1  4  7
b  2  5  8
c  3  6  9
``````

If you need a copy instead, use `to_numpy(copy=True)`.

### pandas >= 1.0 update for ExtensionTypes

If you're using pandas 1.x, chances are you'll be dealing with extension types a lot more. You'll have to be a little more careful that these extension types are correctly converted.

``````a = pd.array([1, 2, None], dtype="Int64")
a
<IntegerArray>
[1, 2, <NA>]
Length: 3, dtype: Int64
Wrong
a.to_numpy()
array([1, 2, <NA>], dtype=object)  # yuck, objects
Correct
a.to_numpy(dtype=‘float’, na_value=np.nan)
array([ 1.,  2., nan])
Also correct
a.to_numpy(dtype=‘int’, na_value=-1)
array([ 1,  2, -1])
``````

This is called out in the docs.

### If you need the `dtypes` in the result...

As shown in another answer, `DataFrame.to_records` is a good way to do this.

``````df.to_records()
# rec.array([('a', 1, 4, 7), ('b', 2, 5, 8), ('c', 3, 6, 9)],
#           dtype=[('index', 'O'), ('A', '<i8'), ('B', '<i8'), ('C', '<i8')])
``````

This cannot be done with `to_numpy`, unfortunately. However, as an alternative, you can use `np.rec.fromrecords`:

``````v = df.reset_index()
np.rec.fromrecords(v, names=v.columns.tolist())
# rec.array([('a', 1, 4, 7), ('b', 2, 5, 8), ('c', 3, 6, 9)],
#           dtype=[('index', '<U1'), ('A', '<i8'), ('B', '<i8'), ('C', '<i8')])
``````

Performance wise, it's nearly the same (actually, using `rec.fromrecords` is a bit faster).

``````df2 = pd.concat([df] * 10000)
%timeit df2.to_records()
%%timeit
v = df2.reset_index()
np.rec.fromrecords(v, names=v.columns.tolist())
``````12.9 ms ± 511 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
9.56 ms ± 291 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
``````

## Rationale for Adding a New Method

`to_numpy()` (in addition to `array`) was added as a result of discussions under two GitHub issues GH19954 and GH23623.

Specifically, the docs mention the rationale:

[...] with `.values` it was unclear whether the returned value would be the actual array, some transformation of it, or one of pandas custom arrays (like `Categorical`). For example, with `PeriodIndex`, `.values` generates a new `ndarray` of period objects each time. [...]

`to_numpy` aims to improve the consistency of the API, which is a major step in the right direction. `.values` will not be deprecated in the current version, but I expect this may happen at some point in the future, so I would urge users to migrate towards the newer API, as soon as you can.

## Critique of Other Solutions

`DataFrame.values` has inconsistent behaviour, as already noted.

`DataFrame.get_values()` was quietly removed in v1.0 and was previously deprecated in v0.25. Before that, it was simply a wrapper around `DataFrame.values`, so everything said above applies.

`DataFrame.as_matrix()` was removed in v1.0 and was previously deprecated in v0.23. Do NOT use!

How to check if any value is NaN in a Pandas DataFrame

How do I expand the output display to see more columns of a Pandas DataFrame?