Alexander
It is better to store everything in UTC
Nikolay
It is better to store everything in UTC
While it's perfectly fine to just store stuff as utc in the database, a user shouldn't be forced to deal with timezone-naive datetimes in applciation code
Nikolay
Especially with timezone-naive dtateimes that are in utc instead of local timezone. It just leads to boilerplate
Nikolay
SQLAlchemy has wonky timezone support where it just silently ignores aware=True for dbs/drivers that don't support it. Automatic conversion to and from naive datetime would be a saner solution and seems to be implemented as a 3rd party library.
Nikolay
#434 seems to be all over the place
It started as a bug report about pony crashing on aware datetimes with obscure error rather than an obvious one. I'm pretty sure pony doesn't always crash, in some cases you will just get unexpected results. The issue now has some hacks and workarounds for specific dtatabses in the comments. Some comments treat lack of timzeone support as a bug, while only lack of clear errors can be considered a bug. You can't support both aware a naive datetime without additional type / additional kwargs.
Nikolay
While it's perfectly fine to just store stuff as utc in the database, a user shouldn't be forced to deal with timezone-naive datetimes in applciation code
Does this sound reasonable at all? I want to have timzone-aware utc times in my code rather that timezone-naive utc or local.
Alexander
I'm not sure I have good understanding of timezone-aware dates. What is the benefit of timezone-aware utc datetimes? User-friendly string representation?
Nikolay
I'm not sure I have good understanding of timezone-aware dates. What is the benefit of timezone-aware utc datetimes? User-friendly string representation?
Mostly being sure that it's actually utc. Python treats naive datetimes as local timezone by default. This has issues on it's own but it becomes worse when naive-utc datetimes get mixed in.
Nikolay
There are several warnings in datetime docs that start with "Because naive datetime objects are treated by many datetime methods as local times, it is preferred to use aware datetimes to represent times in UTC."
Nikolay
Ofc this should be an opt-in since aware datetimes are incompatible with naive ones, you have to convert one of them before you can do something
Alexander
Some time ago I thought about adding timezone support and read some articles about it. If I remember correctly, their conclusion was it is better to have datetime in database as utc without timezone, and convert it to timezone-aware datetimes in application code
Alexander
So, if we add support of timezone-aware datetimes, how should it like? And what Pony should do if a user mixes aware and non-aware datetimes?
Nikolay
it can be a separate type / a kawrg for datetime. Such column should always return aware datateime. I'm not sure if it should explicitly reject naive inputs or if it should convert them.
Nikolay
I guess i would prefer if native types were used for dbs that support them, but I did not read up on possible drawbacks, and it could be considered an implementation detail. You have too look at the db directly to see it.
Nikolay
It seems some databases preserve the timzeone - when you fetch the value you get the same tizmeone that was used to insert it, buuut postgres is not one of them so I personally don't care about this part.
Nikolay
It seems some databases preserve the timzeone - when you fetch the value you get the same tizmeone that was used to insert it, buuut postgres is not one of them so I personally don't care about this part.
Somewhat related is this issue where a user complains about returned time zone being server configuration dependant. I guess I can see this being an issue if that timezone happens to be a variable offset timzeone.
Alexander
This is an article about timezone-aware timestamps in PostgreSQL https://phili.pe/posts/timestamps-and-time-zones-in-postgresql/ And SQLAlchemy author's comment about it: https://news.ycombinator.com/item?id=19533216 """ I completely disagree with this advice: > However, when you come across timestamp, you can’t possibly know in what time zone the timestamp is just by looking at it. yes I do, it's in UTC, because I assume the application is written to correctly handle datetimes. > It is merely a wall time. Maybe it’s UTC, no, it's UTC. > or maybe the developer stored it in local time. right, maybe the application was not programmed correctly, that is always a possibility. But if they are using the timestamp with timezone datatype, it is 100% more likely they have made mistakes in doing so because it is much more confusing to have to fight against PostgreSQL's assumptions and implicit conversions, as the complexity of its behaviors and decisions using this datatype are described in this article. Does the application come up with the wrong answer if the server it runs on, or the webserver container, is mistakenly set to Eastern Timezone rather than UTC? Is the application ensuring that it knows the correct timezone of timestamps as they enter the application, and if so, why isn't it converting to UTC at that moment? Is the application otherwise receiving datetimes of arbitrary timezone, letting them move around business and display logic in their raw form, and waiting until they enter the database to be correctly normalized? that's too late. A correctly coded application does explicit normalization of datetime values the moment they enter the boundaries of the application, just like it would also convert encoded bytes into a unicode-aware strings up front, so that within the application there is no ambiguity, both when persisting and when handling. """
Nikolay
I mostly just don't want to have to implement the conversion to/from aware datetimes myself.
Nikolay
Since I'm not passing time as unix timestamps around, if i have a datetime in my code i want it to be aware specifically so i don't have to worry about whether or not it was converted to utc.
Alexander
I suspect that different people have widely different expectations about supporting of timezone-aware timestamps. Some may expect that database keep non-aware timestamps in UCT as Mike Bayer suggests in the above comment. Others may expect it uses timezone-aware database type. The same in Python, some users may expect they can assign datetimes in any timezones, not only UTC, and ORM makes some proper conversions (or even keep the timezone as it was provided) So it seems to me it may be hard to implement support of timezones which cover all expectations. What you want is only part of this functionality (attribute should accept only timezone-aware datetime values in UTC timezone only, reject all other timezones and local datetime values, and keep information in non-aware UTC column), but will it satisfy other users who want timezone support? I suspect that proper solution is way more complex than may be expected
Nikolay
I don't mind pony accepting naive datetimes in aware columns, if it treats them as local timezone.
Nikolay
If that seem like heresy to the authors it should just reject them
Alexander
Ok, so maybe the simplest non-error-prone way is: - add option tz_aware=True to datetime attributes - reject naive datetimes for such attributes - at the moment of assignment convert datetimes to utc - in the database store it to non-tz-aware timestamp column in utc - return it as tz-aware datetimes in utc timezone
Nikolay
And it should work for any db provider. Now that i think about there are also queries with operations on such aware columns.
Alexander
If all datetimes are converted to utc internally at the first moment then probably we can use current implementation of operations, just don't allow mixing aware and non-aware attributes in the same expression
Alexander
It seems that tz-aware database columns may be tricky and error-prone, and many experienced developers recommend to avoid them
Nikolay
It seems that tz-aware database columns may be tricky and error-prone, and many experienced developers recommend to avoid them
Imagine this: you store utc time in naive column. You apply astimzone to it to later extract date from it in a specific timezone. The server for some reason does not use UTC time and you get an unexpected result.
Nikolay
I don't see how adding "oh yes this IS intented to be utc, you don't have to assume anything" can make things to worse
Nikolay
If we do conversion in Python and use naive timestamp in database it should not be a problem
In that scenario the .astimzone(some_timzeone).date happens in the database
Alexander
In that scenario the .astimzone(some_timzeone).date happens in the database
If database uses naive time it should not be a problem in that use-case
Alexander
if values are stored in utc
Nikolay
If database uses naive time it should not be a problem in that use-case
How does it not become a problem of the time is stored as utc but server is not configured to use utc?
Alexander
If it is a naive date, does it matter how server is configured?
Nikolay
Unless you add some extra steps that ensure the serevr treats value as utc regardles of it's config, at which point you coudl eprhaps use an aware column
Nikolay
If it is a naive date, does it matter how server is configured?
What should result of '2020-05-05 22:00:00'.astimzeone('UTC+3') be?
Alexander
I think server configuration is important for timezone-aware database columns only
Nikolay
For postgres it's kidna both i guess, but the timzone used when returning aware times is a session variable.
Nikolay
It does default to server one but you can change it as session level
Alexander
If that value was read from database column, we assume it is in utc, because we always convert values to utc before saving them to database. Then we can typecast it to utc time inside expression and then convert to desired timezone
Nikolay
It does default to server one but you can change it as session level
And if datetime is aware it's not immediately critical that it's returned in some random timezone. Which is not the case with naive datetime.
Nikolay
If that value was read from database column, we assume it is in utc, because we always convert values to utc before saving them to database. Then we can typecast it to utc time inside expression and then convert to desired timezone
So you have to add a typecast to utc in every expression if you don't want to depend on server configuration. What are the chances it will actually be converting a naive type to aware one in that expression?
Nikolay
I can see how aware types can lead to developer errors if they are actually used to store values with arbitrary timezones. But if they store just utc and conversion is handled by the library and in python it should be pretty safe.
Nikolay
I'll admit I only ever worked with postgresql and mssql aware datetimes
Alexander
I don't remember all the details, but still think that timezone-aware database columns are pretty error-prone. Maybe look up it tomorrow
Nikolay
I don't remember all the details, but still think that timezone-aware database columns are pretty error-prone. Maybe look up it tomorrow
I assume half of those errors are human errors and other half only happens if you use something other than utc, probably variable offset tzs.
Nikolay
I assume half of those errors are human errors and other half only happens if you use something other than utc, probably variable offset tzs.
And probably good chunk of those human errors happens because their programming language doesn't have timezene-aware datetime type or they just didn't use it.
Nikolay
This is an article about timezone-aware timestamps in PostgreSQL https://phili.pe/posts/timestamps-and-time-zones-in-postgresql/ And SQLAlchemy author's comment about it: https://news.ycombinator.com/item?id=19533216 """ I completely disagree with this advice: > However, when you come across timestamp, you can’t possibly know in what time zone the timestamp is just by looking at it. yes I do, it's in UTC, because I assume the application is written to correctly handle datetimes. > It is merely a wall time. Maybe it’s UTC, no, it's UTC. > or maybe the developer stored it in local time. right, maybe the application was not programmed correctly, that is always a possibility. But if they are using the timestamp with timezone datatype, it is 100% more likely they have made mistakes in doing so because it is much more confusing to have to fight against PostgreSQL's assumptions and implicit conversions, as the complexity of its behaviors and decisions using this datatype are described in this article. Does the application come up with the wrong answer if the server it runs on, or the webserver container, is mistakenly set to Eastern Timezone rather than UTC? Is the application ensuring that it knows the correct timezone of timestamps as they enter the application, and if so, why isn't it converting to UTC at that moment? Is the application otherwise receiving datetimes of arbitrary timezone, letting them move around business and display logic in their raw form, and waiting until they enter the database to be correctly normalized? that's too late. A correctly coded application does explicit normalization of datetime values the moment they enter the boundaries of the application, just like it would also convert encoded bytes into a unicode-aware strings up front, so that within the application there is no ambiguity, both when persisting and when handling. """
First part of zzzeek's comment is "db datetime values are utc or it's a bug". Sadly, that's just not true in the real world. Now, if it's an aware type, effectively equivalent to a unix timestamp, and that unix timestamp is wrong, then it's definitely a bug. The part about PostgreSQL conversions only maybe applies to values being read. If you pass an aware datetime to a db driver, and it's used as aware datetime in the db, then there's no room for error, save for bugs in the driver/database/python itself. And even values being read should be correctly returned as timezone aware datetimes by the driver. Arithmetic on aware types is not that different from arithmetic on unix timestamps, the article is mostly "there are timezones with variable utc offset". Perhaps it's still worthwhile for pony to set session timezone to utc in case of postgres, and to convert values to (aware) utc in case of dbs that do store tz information, just to reduce the chances of database/driver bugs, remove possible issues with some specific timezones. There will probably be some people unhappy about it, but if the alternative is no tz support I'm sure they will be willing to compromise. Just tell them there will probably be an opt-out flag eventually. Well, at the end of the day I'm happy enough if I just get correct timzeone-aware datetimes form the orm without having to write the conversions myself. At least for now. I'm not sure if using naive types in the db can cause issues (e.g. with operations on them) / has drawbacks aside from semantics mentioned at the start of this message. But I imagine it makes more sense to do timezone-related operations on timezone-aware types.
Lucky
If we do conversion in Python and use naive timestamp in database it should not be a problem
Yes. In fact, why not updated_at=orm.Optional(datetime, default_timezone="GTM") Or something? Store that thing in UTC, convert it back on retrieval. As long as that python datetime object knows it's UTC you can easily change it to every timezone one need, and as proposed, you could even set a default time zone. Like we use a ORM to not deal with the database, so I wouldn't even care if in the database everything is stored with a text field and the epoch in unary encoding as banana emojis instead of an regular UTC timestamp fields.
Lucky
1970-01-01 00:00:00 = "" 1970-01-01 00:00:01 = "🍌" 1970-01-01 00:00:02 = "🍌🍌"
Lucky
Ah yes, I just invented BananaORM
Alexander
Then 1970-01-01 00:00:02 = "🍌🍌" Should be 1970-01-01 00:00:02 = "🍌 "
Lucky
No, the whitespace is not the opposite of a banana. Try 🍌/🐒 for 1/0.
Lucky
If there's a monkey there will be no banana
Jim
😂
{°•.~.•°🇦 🇻 🇮 °•.~.•°}
Hi guys, does anyone know tools for export and import from excel / csv files?
J J
I've used tableplus
Lucky
Let me 🦆 DuckDuckGo that for you: 🔎 python csv
Lucky
import csv >>> with open('eggs.csv', newline='') as csvfile: ... spamreader = csv.reader(csvfile, delimiter=' ', quotechar='|') ... for row in spamreader: ... print(', '.join(row)) Spam, Spam, Spam, Spam, Spam, Baked Beans Spam, Lovely Spam, Wonderful Spam https://docs.python.org/3/library/csv.html
J J
I thought he meant to database
J J
🤷‍♂️
Lucky
I mean, that's just for row in reader: SomeTable(row[0], row[1], etc…)
Lucky
Holy crap, ever noticed that small button in the python docs? It hides the ">>>" prefixes as well as the examples returns for easy copy.
Jim
Nice top :-)
Henri
Is there a way to create a table with tree structures using the materialized path pattern with PonyORM? PostgreSQL has the LTREE extension which provides the Ltree column type and allows also optimized indexing with a GiST index. Though I don't think this is compatible with PonyORM. Any ideas?
Alexander
Pony doesn't have direct support of LTREE, but probably you still can use it with raw SQL. Something like that: class MyEntity(db.Entity): path = Required(str, sql_type='ltree') ... select(x for x in MyEntity if raq_sql(''' x.path <@ '001.003' '''))
Henri
Ok so I can use sql_type for the entity column. That sounds promising.
Henri
And is there a way to set the GiST index in Pony or must I set it manually?
Alexander
You need to set it manually
Henri
Ok
Henri
But I think that is not a big deal...
csa3d
what's the best way to completely replace a single table in an existing db