BUG: Timestamp lose precision (ms)

[PrimozGodec]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

>>> import pandas as pd

>>> t = pd.Timestamp("01:01:01.01").as_unit('ns')
>>> t
Timestamp('2023-05-08 01:01:01')  # it seems ok

>>> t.timestamp()
1683507661.0  # lost milliseconds

>>> pd.Timestamp("01:01:01.01").as_unit('ns').timestamp()
1683507661.0  # still lost milliseconds

>>> pd.DataFrame([[pd.Timestamp("01:01:01.01")]]).dtypes
0    datetime64[ns]
dtype: object

>>> pd.DataFrame([[pd.Timestamp("01:01:01.01")]])
# no milliseconds also
                    0
0 2023-05-08 01:01:01

Issue Description

I know that Pandas 2 uses seconds as a default unit: #52653; I think something goes wrong when I want to more precise time. Look at the example; milliseconds are lost even when I use as_unit. Also, when converted to data frame type suggest that precision is ns but does not show milliseconds.

Am I doing something wrong or is it a bug?

Expected Behavior

I think milliseconds should be in timestamp at least when precision is set to ns by as_unit.

Installed Versions

INSTALLED VERSIONS

commit : 37ea63d
python : 3.10.10.final.0
python-bits : 64
OS : Darwin
OS-release : 22.4.0
Version : Darwin Kernel Version 22.4.0: Mon Mar 6 20:59:28 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_T6000
machine : arm64
processor : arm
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.UTF-8
pandas : 2.0.1
numpy : 1.24.3
pytz : 2023.3
dateutil : 2.8.2
setuptools : 67.7.2
pip : 23.1.1
Cython : 0.29.34
pytest : None
hypothesis : None
sphinx : 6.1.3
blosc : None
feather : None
xlsxwriter : 3.1.0
lxml.etree : 4.9.2
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.12.0
pandas_datareader: None
bs4 : 4.12.2
bottleneck : 1.3.7
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.7.1
numba : None
numexpr : None
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.10.1
snappy : None
sqlalchemy : 1.4.47
tables : None
tabulate : 0.9.0
xarray : None
xlrd : 2.0.1
zstandard : None
tzdata : 2023.3
qtpy : 2.3.1
pyqt5 : None

[mroeschke]

timestamp() is defined to return a POSIX timestamp which is number of seconds since Jan 1 1970, so this is the expected behavior: https://docs.python.org/3/library/datetime.html#datetime.datetime.timestamp

Closing as a usage question

[PrimozGodec]

@mroeschke, thank you. It is understandable and maybe a bad example from my side. The problem I have is not the use of timestamp() (it was just used as a demonstration) but the following case.

>>> df = pd.DataFrame([[pd.Timestamp("01:01:01.01")], [pd.Timestamp("01:01:01.02")]])
>>> df.dtypes
0    datetime64[ns]
dtype: object
>>> df[0].astype(np.int64) // 10**6
0    1683594061000
1    1683594061000
Name: 0, dtype: int64

The example creates a data frame from Timestamps that include milliseconds. Then I pack them in DataFrame, and they lose milliseconds (dtype still seems to the point that precisions are ns). Lost of milliseconds is seen when I transform the column to int and then divide by 10**6 to convert from ns to ms. The timestamp 1683594061000 represents "01:01:01" (and not "01:01:01.01", which was the initial time). The last three digits represent milliseconds and should be 010 if milliseconds from the string would be preserved. Is it a bug, or should I convert it to DataFrame differently? In pandas<2 milliseconds were preserved.

[MarcoGorelli]

thanks for the report, but I can't reproduce this

In [21]: >>> df = pd.DataFrame([[pd.Timestamp("01:01:01.01")], [pd.Time
    ...: stamp("01:01:01.02")]])

In [22]: df[0].astype(np.int64) // 10**6
Out[22]:
0    1683594061010
1    1683594061020
Name: 0, dtype: int64

[MarcoGorelli]

can't even reproduce the original example

In [23]: pd.__version__
Out[23]: '2.0.1'

In [24]: >>> t = pd.Timestamp("01:01:01.01").as_unit('ns')
    ...: >>> t
    ...: Timestamp('2023-05-08 01:01:01')  # it seems ok
    ...:
    ...: >>> t.timestamp()
Out[24]: 1683594061.01

how did you install pandas?

[PrimozGodec]

I installed the latest (2.0.1) version of Pandas from PyPI. the same also happens on the main branch version of this repository. My OS is MacOS (arm). Python version 3.10

[MarcoGorelli]

@phofl can you reproduce this by any chance?

[MarcoGorelli]

@PrimozGodec could you try in a fresh virtual environment please, in which you only install pandas?

[PrimozGodec]

I have already tried it in a fresh environment. Now I tried with all Python versions (3.8 - 3.11). Same behaviour in all four versions on MacOS. I can try tomorrow on Windows/Linux too.

[PrimozGodec]

I can confirm that it also doesn't work correctly on Windows and Ubuntu.

Another thing that I noticed and may help is:

>>> pd.Timestamp("01:01:01.111").value//10**6
1683680461000

miss milliseconds.

[MarcoGorelli]

not sure what to do here, given I can't reproduce, but investigations would be welcome

>>> pd.__version__
'2.0.1'
>>> import pandas as pd
>>> pd.Timestamp("01:01:01.111").value//10**6
1683680461111
```

[PrimozGodec]

I can try to debug. It will probably take a while for me to figure things out since I am not fluent in debugging cython code.

[DeaMariaLeon]

I can't reproduce the bug either, but my mac is old @PrimozGodec.
Everything else is the same.

[asishm]

>>> import pandas as pd
>>> pd.__version__
'2.0.1'
>>> t = pd.Timestamp("01:01:01.01").as_unit('ns')
>>> t
Timestamp('2023-05-12 01:01:01')
>>> t.timestamp()
1683853261.0
>>>pd.Timestamp("01:01:01.111").value//10**6
1683853261000

this is from a fresh conda env with py3.10 and pandas 2.0.1 on WSL. trying to get main built - but having some issues following the instructions after the switch to meson build

[jbrockmendel]

closed by #54097