I’ve been reading Mastering Regular Expressions by Jeffrey E.F. Friedl, and since nobody in my life (aside from my wife) cares, I thought I’d share something I’m pretty proud of. My first set of regular expressions, that I wrote myself to manipulate the text I’m working with.
What’s I’m so happy about is that I wrote these expressions. I understand exactly what they do and the purpose of each character in each expression.
I’ve used regex in the past. Stuff cobbled together from stack overflow, but I never really understood how they worked or what the expressions meant, just that they did what I needed them to do at the time.
I’m only about 10% of the way through the book, but already I understand so much more than I ever did about regex (I also recognize I have a lot to learn).
I wrote the expressions to be used with egrep and sed to generate and clean up a list of filenames pulled out of tarballs. (movies I’ve ripped from my DVD collection and tarballed to archive them).
The first expression I wrote was this one used with tar and egrep to list the files in the tarball and get just the name of the video file:
tar -tzvf file.tar.gz | egrep -o '\/[^/]*\.m(kv|p4)' > movielist
Which gives me a list of movies of which this is an example:
/The.Hunger.Games.(2012).[tmdbid-70160].mp4
Then I used sed with the expression groups to remove:
- the leading forward slash
- Everything from
.[to the end - All of the periods in between words
And the last expression checks for one or more spaces and replaces them with a single space.
This is the full sed command:
sed -Eie 's/^\///; s/\.\[[a-z]+-[0-9]+\]\.m(p4|kv)//; s/[^a-zA-Z0-9\(\)&-]/ /g; s/ +/ /g' movielist
Which leaves me with a pretty list of movies that looks like this:
The Hunger Games (2012)
I’m sure this could be done more elegantly, and I’m happy for any feedback on how to do that! For now, I’m just excited that I’m beginning to understand regex and how to use it!
Edit: fixed title so it didn’t say “regex expressions”
Good job !
I highly recommend trying out the various online regex editor.
These WISIWIG kind of editors are great because you immediately see what the regex is catching and for what reason.
I took the first one in my search results but try different ones.
Also I used GPT to get some regex for some specific strings and it can be helpful to get a quickstart at building a specific regex.
In that case I was building a regex for a specific log from postfix.
PS: just make sure to select the correct flavor of regex you are using in these online tools.
Edit: Also one of my favorite YT channels has pretty cool videos on RegEx : https://youtu.be/6gddK-cOxYc?si=0bnNkSDzifjdxwjU
Piggybacking onto this to mention my go-to online RegEx editor: RegExr. It lets you test the regex as you type, explains the particular symbols used, as well as has a sidebar where you can see different pattern types categorically. I’ve been using it for almost 2 years now, and haven’t had any reason to use much else (after I discovered this).
I stumbled upon this regex crossword puzzle a while back. I was never good enough to get it, but it seems like it could be fun.
Just adding my congrats. Good job, OP. Regex is super useful stuff.
It is a great book, although a bit outdated. In particular, nowadays
egrepis not recommended to use.grep -Eis a more portable synonim.Some notes on you script:
-
You don’t need to escape slashes in grep regex. In the sed
s///command better use another character like##so you also can leave slashes unescaped. -
You usually don’t need to pipe
grepandsed,sed -nwith regex address and explicit printing command gives the same result asgrep. -
You could omit leading slash in your
egrepregex, so you won’t need to remove it later.
So I would do the same with
tar -tzvf file.tar.gz | sed -En '/\.(mp4|mkv)$/{s#^.*/##; s#\.\[.*##; s#[^a-zA-Z0-9()&-]# #g; s/ +/ /g; p}'nowadays egrep is not recommended to use. grep -E is a more portable synonim
Not directed at you personally, but this is the kind of pointless pedantry from upstream developers that grinds my gears.
Like, I’ve used
egrepfor 25 years. I don’t know of a still relevant Unix variant in existence that doesn’t have theegrepcommand. But suddenly now, when any other Unix variant but Linux is all but extinct, and all your shell scripts are probably full of bashisms and Linuxisms anyway, now there is somehow a portability problem, and they deem it necessary to print out a warning whenever I dare to runegrepinstead ofgrep -E? C’mon now … If anything, they have just made it less portable by spitting out spurious warnings where there weren’t any before.
-
Just to chip in because I haven’t seen it mentioned yet, but I fing LLMs like ChatGPT or Microsoft Copilot are really good at making regexes and also at explaining regexes. So if you’re learning them or just want to get the darned thing to work so you can go to bed those are a good resource.
