#435 Stop with .folders in my ~/
25:34 Share

Hello and welcome to Python Bytes where we deliver Python news and headlines directly to your earbuds. This is episode 435, recorded June 9th, 2025. And I am Brian Ocken. And I'm Michael Kennedy. And this episode is sponsored by us. So please check out all the awesome courses at Talk Python Training and lots of different options for learning PyTest at Python.

python test.com. Um, so check that out. And also thank you to our patron supporters. They've been sticking around for a long time and we really appreciate them. If you'd like to connect with us more about this later, but if you'd like to connect to it with us, um, uh, please check out the links in the show notes, um, links to, uh, both blue sky and mastodon are, um,

are there and if you would like to join us live, you can head on over to pythonbytes.fm/live and it'll show you how to do that. There's a link at the top. It's really easy. Then also, when you're listening and you want to check out all the cool things that we talk about but don't want to write them down, no worries. Just go ahead and sign up to be part of the newsletter and we'll send those directly to your inbox. With that,

Do you have a cool item for us, Michael? I do. I have some stuff I want to talk about. It's an update on things I've spoken about before, but it could have been six years ago. I can't remember when we covered this, and I, for some reason, didn't do a search. So here's the deal. You are writing an application, and you're doing things like, hey, I need to have my app

save some content in a file. I don't want the user to access it. I want my app to be able to get to it. And that's pretty much it. Like maybe a SQLite DB file, right? That's not a thing that the users should be tracking down. You don't want to put it in their My Documents on Windows or somewhere in the user profile on Mac or Linux or something like that, right? You just, you need to save it somewhere. It needs to be associated with the user. They need to have permissions right there. So I previously talked about

about appders, the appders package, a small Python module for determining appropriate platform specific directories like user directories or stuff like that, right? So where's your config? Where's your cache directory? So on Mac OS, that's like tilde slash capital library, capital L library.

cache, then you probably use app name, something like that. Those kinds of things, right? You want to get access to that and put things in there. And it drives me nuts. Brian, I go to my user profile. If I want to look at hidden files, there's a ton of dot this app dot that app settings in my user profile. And they're not there for me. They're there because the designers of those apps were too lazy usually.

too lazy to put that into the appropriate location on that platform. Right? There's some that maybe I should mess with, like for example, oh my Z shell, you might want to go and tweak like your, your, a prompt, how that looks or whatever. And right. You would get to that from there, but a lot of things they just do it. Cause like, well, we'll just do a dot cause it's hidden. People won't see that. Just drop it right in there. So I implore people listening to put it.

To put stuff in the right place or a better place. Well, okay, I'm guilty of this, but I don't really think about Windows too much. I just think about like in a Linux-y environment, the home directory is usually the right place. Well, maybe. I mean, let me see what I got in here. I don't think so, but let me do an A. Okay.

Where do I do my hidden ones there? You know what? I'll do it this way. Hold on. I'll do it this way. So, for example... Well, I'm probably doing it the wrong way there, too. So, you're probably right. Well, maybe. But there's, like, Studio 3T, which is a MongoDB GUI management tool. It has all of its settings in here. Is that something that I should be working with? Should I be seeing this? Probably not. It's, like, a .3T folder, and it's got, like, my roaming history and crap in there. I've got .Android for whenever I have the Android SDK doing something. I've got...

I mean, I can see like I've got a .aws. Maybe it's got things in there, right? But I have a .mapplotlib. For real, there's a .mapplotlib in here.

Why? Why is it here? I don't know. There's a pip audit cache. Literally, it's called cache. Should I see the cache? No, the cache is not for reading. It should be in, there's literally a cache folder on all these platforms where it goes. And maybe on Linux, it literally is just like there, but okay, fine. But on the other ones, it shouldn't be, right? Anyway, maybe I'm ranting a bit hard on this, but you should probably download

do these correct things. And I wasn't even thinking about Windows, although Windows does have a different set of places things go, right? So this thing was great and I recommended Apters, but if you go up a little bit, it says, "Note, this project has officially been deprecated." - Oh no. - No, we were so close.

So it says you may want to check out PlatformDirrs, which is my recommendation. So moving over to PlatformDirrs, this is under toxdev, T-O-X dev. So cool group there. And it's a small Python module for determining the appropriate platform specific directories, such as I just said. However, it's better in some ways.

So it has, I don't know if you noticed before, but there's maybe three or four directories on the app. This one has documents, downloads, pictures, videos, music, desktop, runtime, et cetera, et cetera. Lots of different places that you might want to go to depending on what platform you're on. Right. It has.

better typing. It has been rewritten to use path as in path objects from path lib and all that. It's got CI testing against all the different platforms that it supports. So very, very cool that this is not only continues to live, but it's actually gotten much better.

Yeah, and the talks development group is going to keep it updated. So, cool. Yeah, exactly. It can't require that much support over time, right? These things are not super dynamic, but I do really appreciate them modernizing it. You know, I think that some of the changes were that basically apptors got out of sync with Python rather than out of sync with the platforms, right? It was just like, well, it doesn't have types. And so now we got, you know, just a bunch of stuff like that.

So I'm very happy to see it and recommend platform DERS. I already used it to massive success on some projects. I was like, I want to use this and some cool caching things. And it made my app so much better. And guess what? There's not yet another dot Michael's app cache in my tilde slash.

Over to you. I'll have to do this on some of my applications. It's worth considering. So I'm glad you brought it up. All right. I'd like to actually want to do a shout out to Bob Belderbos because...

He posted, I think he posted on several platforms, but I saw it on LinkedIn. It says, I love make files. They save me time and help my team work in more uniform way. I'd love to be able to say this. I love make files also, or actually I don't love them. I'm just used to them. So I throw in a make file into something that makes sense. But for a very long time, nobody else on my projects have been familiar with make files. So I can't. That wouldn't be me. I'd be like, what's a make file? Why do we have this?

that we do in Python? So some Python projects I use talks because you can do kind of some similar stuff with talks, but it is annoying that it creates a virtual environment. It's better with UV now because it's fast. But still, there is a new kid on the block, or at least new to me, that Bob has introduced, and it's called Poe the Poet.

And I was a little leery because I thought it maybe was too tied to the poetry project. But even if it is, you don't have to use it with it. I'm using it without it, of course. But anyway, so the thing I love about this is, as an example, I'm leaving this on Bob's post because it's great. In your pyproject.toml, that's where you define the textboxes.

the tasks. And this is great because I don't have to have an extra thing and an extra file in my project. This makes sense to just put it there. So he has an example of like, so a test action would be, you can just define it as PyTest or something or Cove. You can have a, the have PyTest running with Cove with Cove.

with coverage. Um, the linter is rough, which is, it's funny. A lot of people still do this, but actually I remember rough better than I remember Lent, um, anymore, but you know, it's just me. Uh, but anyway, some cool, uh, um,

type. Oh, he has tie check for, for the T Y. What? Wow. He's on top of that. That is super new. Um, integrates well with UV. I tried it out. It is pretty slick. So the, uh, Poe, the poet documentation, uh, says, uh, it's a batteries included task runner that works well with poetry, um, or with UV. I'm definitely using it with UV. Uh, the thing that there's a bunch of cool things, how to define this. So in the, in the,

the documentation is pretty good. So you can, like, like Bob said, you just add a, a, a section called tool.po.tasks and, and then define a bunch of actions. There's ways to have, like built up actions, like you would with a make file or talks have, actually, I don't know if you can do this. Anyway, you'd have multiple actions that run one after another. The sequences is pretty cool. You can also add help text to, to things, which is,

which is nice anyway. Um, but having it right in, I just love it. It's right in the pipe product at Tomo and it works well with UV. Um, love that. So, uh, definitely going to switch it for a lot of projects. And there's an example, like I'm not sure where the example is right now, but in the kind of getting started, um, or maybe it was Bob anyway, uh, adding it to, uh,

adding it to your development dependencies within your PyProject.toml is slick because then now if you're using, hopefully you're using development dependencies within your project.toml so that new developers can just

install all the tools they need to just by doing using that so pretty cool love it yeah yeah very cool and by the way did you catch that the the logo of Poe the Poet is Edgar Allan Poe yeah he's the poet very interesting okay next up is some data science there's an article I'm linking to on the new stack by

Jacob's, uh, Jake, uh, Jackson rather. Sorry. And this is that hand does 3.0 will significantly boost performance by replacing numpy with pie arrow as the default engine, enabling faster loading and reading of columnar column and

So since 2.0, I think Pandas has had possible support for PyArrow, which is more of an analytics type backend than NumPy, which is more database-y, like traditional rows. Like here's an entry in its internal structure, making it easier

much easier to ask questions like what's the average or the max of this aspect of the data, right? What is the average sales price for these million rows is way faster in a columnar orientation because you just go right down the data as it is. It makes it slower for row type of operations, but that's less common.

right for data science it also does much much faster reading and writing of files and all sorts of things so i recently spoke with reuben lerner about the pyar revolution on talk python not too long ago and that was a lot of fun basically we talked about all the benefits of pyar and why you might consider it but the news here is

that this will be the default in 3.0 for Pandas. When is that coming out? Who knows, right? Pandas has been around for like many, many, many years and it's on version two. So maybe it's gonna take, what, another 15 years for version three to come out? I don't know. - Probably not. - Versions are, I think the perception of versions is changing.

over you know we talked about zero over and all this stuff but i think versions are starting to go faster these days right jango is on five and it was just on four and it's been around a long time but it should be out pretty soon so i'm not exactly sure on the timing i know some people will know better than me but anyway also may have been like that just have switching out the back end is a big enough change that they decided that's a good reason to switch to a

Yeah, that's a very good point. It says, over time Pyro is becoming better and better integrated with pandas, but using it as a backend is still experimental and isn't recommended in production, presumably until 3.0 where it becomes the default in which it would become recommended because you'd have to work around it to know better, right? Anyway, that's my item. Check out Pyro. Check out the conversation I had with Reuven about it to see what's coming for pandas. Cool. All

All right. I wanted to talk about data science a little bit as well. So, and I'm going to point to a project called Point Blank. It's a data validation made beautiful and powerful. At least that's the sales pitch or the sub point. Anyway, so why I don't do a lot of data science, but I really think this is a cool idea. Even without data science, any sort of pipeline stuff of data

checking data within your pipe

within your pipeline makes sense and what to do with it. So here's the idea. So it says it's a powerful yet elegant data validation framework for Python that transforms how you ensure data quality. There's a chainable API that you can use to validate data against comprehensive quality checks. And it says visualize through stunning interactive reports, which I haven't looked at the reports yet. But anyway, so just at the top example,

you've got you import point blank and then you have these validation features where you can take a data set and invalidate it. In this example, it says, okay, on column D, there's validate that the values are greater than 100. There's more than 100 values there. And then actually on column C, validate that there's less than or equal the values

or the values are less than or equal to five. And then there's some, that the, there's certain columns exist, make sure the date and the time exist in those columns. So this, you get, you get data frames in and, or data sets and data frames, and you got to be able to make sure that they're in the right shape, the data's in the right place. And, and that's awesome. And what do you do if they don't? Well, there's, there's a REPL with some validation, get tabular report. That's

That's great for when you're debugging stuff and looking at it. But what about in real time? You're not going to want to do that. And this is something you can use in real time to check all this because it's got these cool actions. So there's a bunch of actions as part of this where you can set up callbacks. It's got a callback.

a default one that's built into notify you on Slack. I don't use Slack anymore, but there's a function called one. If there's any failures, you can set up your own notify function, which I love this idea of just like give me an API to fill this in and I can fill that in with whatever I want. I can call it rest API. I'm going to send myself an email or whatever. Yeah, whatever.

I'll hook it up. And so this idea of like, just run this all the time. And then if there's problems with your data chain, let somebody know about it in real time.

Love that idea. You can even log it. Lots of stuff there. And looks like the interface is pretty intuitive. They've got a roadmap for additional validation methods, advanced logging capabilities, more messaging actions. So kind of a neat project. I love that it's set up already with a code of conduct and a governance system. So pretty promising little project.

So that's it for that. I really did. I like the idea of this a lot. Yeah, I like it too. I'm not sure it'd be fun to hear somebody that's used it and see is it something that really can be used in real time? Does it slow things down too much? I imagine not because you're going to do a little bit of work just after you've done a lot of work in a few places, validate in a few places in the pipeline to make sure everything's still kosher. Yeah.

It definitely, you know, data science has the feel a lot of times of I'm going to do a little bit of work to get things set up and I'm going to do some complex calculations on lots of data. Yeah. Rather than a bunch of tiny little function calls. I think here's a function, a bit of functionality I'm adding to like an API. It would be totally reasonable to say I want that to be under 10 milliseconds for this API call. No one's going, I need my notebook to process stuff in less than 10 milliseconds. Like, yeah.

If you add a little overhead, it's probably fine. This certainly looks like it's going to be useful for unit tests as well, right? Yeah. Yep. And checking to make sure-- yeah, and that's the sort of thing is like, OK, you've checked all the pieces. But this is-- once you put everything together,

Are we like losing something somewhere? Somebody is some part of the process filling in nulls or, or treating, treating invalid data incorrectly and putting garbage in there or something. Um, um, yeah, it's super easy to lose pieces of data on some of these things. How is this for a tie in on our, onto my first extra. Okay. If you're working with polars or something, you can say, I want to do some calculations. And what you get back is a new data frame by default, what you get back are the

or the columns you've asked for, like the new computer columns. But maybe what you intended is I want to add a column to the existing data frame, not take a data frame and transform it into a new data frame that has just the stuff you asked for. So instead of doing an operation, you would say dot with columns.

then do the operation and it'll keep the original data frame and then add on to it, right? You would maybe like with your, that these columns exist test, then you would be able to say, oh, we've lost some of the columns. Who forgot the width columns? Why would you know to do that, Brian? Because you take our Polar's for Power Users course that we just released last week. Oh, nice. Before, yeah. So this is out, Spike, yeah.

Chris Trudeau, it's really awesome course, Pollers for Power Users. Three hours, almost, just under three hours, and you can go and learn a whole bunch of cool things, how to work with pollers, and pollers uses PyArrow underneath the covers, or Arrow underneath the covers, so you can also get some cool experience with that. So people, check this out. It's been well-received and continuing to go. All right, that's my only extra I have. I have a joke, but got any extras before we do that? Oh, I'm loaded with extras this week, so...

Let's load them up. So there's my point blank. I was interested. So I'm, I'm, I'm you, I haven't been using rough. I use rough a lot, but I don't tweak it very much. I haven't pulled, there's a bunch of new rules or not really new, but there's a lot of rules that you can turn on for rough for, for checking your code. And the, the, if you look at the rules list, there's a lot, there's a

there's a lot of rules that you could add. So I was interested in this discussion on, on Reddit about, Hey, rough users, what are the rules you're using and what are you ignoring? Um, this one was amusing to me that said, I start every project and select all and ignore none. Um, that's not realistic because some of them, uh, conflict with each other. Uh, so, uh, I think this was probably a joke, but, but I'm not sure. Uh,

I do like, so there's some serious people that said, hey, this is my set. It had like PyFlakes rules and PyCode style and warnings and errors. Seems like a decent set. It's just kind of a fun thing to see what other people are using to maybe check out some things. Here's somebody else that has a different way. Instead of picking which rules they use, it's a select all but ignore certain rules

certain rules that don't make sense for them. And yeah, just kind of an interesting discussion. Anyway, I'm not sure if I covered this one already, but this was as of May 26th, Python 3.14.0 beta 2 is out. So if you're testing new Python versions, make sure to keep up on the new betas that come out.

Nice. Yeah, I don't know. I don't think we covered it. Okay. And I guess sort of a, oh, I got a couple more topics. One of them is if you happen, like just randomly wanted to switch Mastodon servers, it's not and not lose anything like lose anybody that might be following you already. There is a way to do it. And we'll link to an article on how to do this on Feddy tips. And it's not,

This is not a trivial action, so I guess don't do it on an empty stomach. Looks like a lot of steps, but I'm glad somebody pointed that out. Why might you? Well, there's been a little bit of drama on Fostadon lately, and I...

I am not the expert on this, but there's enough people I trust that are migrating away from FOSDON that I'm looking into it. I just haven't done it yet because, yeah. Yeah.

Anyway. Thoughts on Fostadon? I have noticed people who I thought followed me. I got notifications on Mastodon that they had followed me. And I think that's why I think they were moving accounts. Yeah. I got some too. I'm like, well, I want to follow them too. And I was already following them. So if you want, and so I guess this is public service as well. Even if you're not considering switching, if you're getting notifications from people that you're like, I thought they all already followed me. Yeah.

it's probably because of something around this. The last thing I wanted to talk about, um, is, uh, I was, I've, I've got some, uh,

I think that my Python test.com, is that what I got? Yeah, that's my site. I don't know if I can trust the analytics that I was using before because I think I'm getting hit hard by AI bots and stuff. So I did update my robots.txt, but I'm also checking out a new analytics package. I'm trying out Fathom Analytics, and I chose them because they don't,

collect data on anybody. It's just a which pages were hit and which pages weren't and they are GDPR compliant so

So yeah. Anyway, I'm not much more of a plug than that other than I'm trying it out. And yeah, we'll see. Yeah. Very cool. That's it. Awesome. Awesome. Yeah. I'm still loving Umami analytics. Again, GDPR, even better than GDPR compliant, like no cookies whatsoever. How's that? And we self-host it. So we don't have to, we don't share the data with anybody. So we don't have to tell you about how we're sharing our data with people, but we're not sharing data with people, which I think is great. Okay.

Okay. Effortless. It says effortless. I'm not sure it's effortless. I've expended effort when the migrations don't work or whatever, but you know what? It's fine. It's still good. It's better than Google analytics. All right. Are you ready for something fun as well? Yes. I'm not sure if I call this fun. I think it's fun E in a sort of sympathy way. So this joke is a cartoon called programming humor from programming here called emotional damage. And I think it's,

Brian, you want to be the one? You want to act this out? You want to be the one that does the first square? Sure. Hey, there's two people looking at a dog. Does your dog bite? No, but it can hurt you in other ways. I don't know who's going to say this. I'll do the dog. The dog says, the feature you spent hours coding will not be deployed. Management finds it unnecessary. And the person starts crying. That's so bad.

No, but it can hurt you. Yeah. Do you know what's more rough than, than, uh, did software having that happen with you? Um, hardware people, um, I've got, I remember this was decades ago, but it was working with a team and somebody got a, was retiring and they were kind of roasting him during the retirement. But, um, uh, he was an FPGA developer and her known ASIC developer doing, uh, um, custom chips. And,

And like he'd worked on like dozens of projects and only like 5% of them ever shipped. Most of them got canceled before they, you know, before the project end date. So that's rough.

Yeah, the lead time is definitely longer in hardware than it is in software, right? Like from the time you work on it until the time it's out. Yeah, it's in like, you know, single digits or fractional digits of years instead of weeks or months. Yeah. I definitely think about the, I'll watch documentaries about, you know, like the Cassini probe that went to, was that Saturn it went to? Yeah.

And these people will be like, yeah, we started that 25 years ago and now I'm done with this project. Moving on to the next. It's like you get two or three projects for your life. Crazy. Those things better not burn up in the atmosphere on their way out. Yeah. Well, may the dogs that talk about unshipped features stay away from you all. Yep. Bye. Bye.