Transforming, sorting, and grouping JSON documents in the command-line - jq tutorial

  Рет қаралды 26,099

Szymon Stepniak

Szymon Stepniak

Күн бұрын

Пікірлер: 90
@szymonstepniak
@szymonstepniak 4 жыл бұрын
⏱️ Timestamps: 00:00 | Intro 00:30 | What we are going to learn? 01:10 | Calling openlibrary.org Search API with curl 02:23 | Using object/value iterator 02:51 | Transforming input JSON to the desired format 03:43 | Filtering null values with the select function 04:23 | Sorting an array by the specific field using sort_by(expr) 05:46 | Limiting the number elements using limit(n;expr) 06:48 | Grouping elements by the specific field using group_by(expr) 09:22 | End screen
@paweosmolski5567
@paweosmolski5567 3 жыл бұрын
This video is one of the best things KZfaq has to offer. Excellent ratio of information per minute. Thank you so much
@szymonstepniak
@szymonstepniak 3 жыл бұрын
Thank you very much for your kind words, Paweł! It's always a very motivating thing to hear. Take care and have a good day!
@MikeCairns1
@MikeCairns1 2 жыл бұрын
I agree. I read through the documentation for jq over and over and tested for hours. This example just made it click and solved a problem that I have been working on all day that had a very simple solution. Amazing job! Thanks for posting!
@jnetto
@jnetto 3 жыл бұрын
Fantastic! The best jq intro in the universe! It's almost as if all the articles, tutorials, docs, and etc, were condensed in 9m44s. Thanks for sharing your knowledge!
@szymonstepniak
@szymonstepniak 3 жыл бұрын
Thanks for your kind words, Jose! I plan to publish a few more videos explaining jq, so stay tuned! Have a good day!
@nilanjansarkar100
@nilanjansarkar100 Жыл бұрын
Every secodnis pure gol. thanks for this wonderful video
@szymonstepniak
@szymonstepniak Жыл бұрын
Thank you so much for your kind words! Take care, and have a good day!
@Hector-Site
@Hector-Site Жыл бұрын
Excellent, concrete and useful examples
@szymonstepniak
@szymonstepniak Жыл бұрын
Glad you like them!
@user-ew6mz8kz8v
@user-ew6mz8kz8v 9 ай бұрын
Очень доходчиво! Спасибо!
@szymonstepniak
@szymonstepniak 9 ай бұрын
Thank you! Take care, and have a good day!
@sriramvelamur
@sriramvelamur 3 жыл бұрын
jq is definitely a godsend; I recently moved parts of my application pipeline to a series of jq processes and life couldn't be peaceful. Thanks for the video, Szymon. With the limit operator, (at least with v1.6), there is another nice way to do it too - "| .[0:n]". Coming from a Python background, this looks easier to comprehend like the slice operation and lesser characters too.
@szymonstepniak
@szymonstepniak 3 жыл бұрын
Couldn't agree more, Sriram! Thanks for the valuable comment. Take care, and have a good day!
@kossboss
@kossboss 8 ай бұрын
Thank you for creating an example with some complex json that’s easy to understand and helps folks learn
@szymonstepniak
@szymonstepniak 7 ай бұрын
Thank you for your kind words! Take care, and Happy New Year!
@conniedave7628
@conniedave7628 Жыл бұрын
Thanks for the FULL examples. Most only show pieces.
@szymonstepniak
@szymonstepniak Жыл бұрын
Glad you liked it! Take care, and have a good day!
@M1k3M1z
@M1k3M1z 3 жыл бұрын
Such a great group of lessons, thank you for making these! They're so well done!
@szymonstepniak
@szymonstepniak 3 жыл бұрын
Thank you for your kind words! Have a good day!
@xGorynychx
@xGorynychx 4 жыл бұрын
Owing to this series, I used JQ to do some cool things in my day-to-day work tasks. Would be great to see next part!
@szymonstepniak
@szymonstepniak 4 жыл бұрын
Working on a script. Coming soon! :)
@karlastambla2788
@karlastambla2788 2 жыл бұрын
Wow, that's jq on steroids. Magic!!
@szymonstepniak
@szymonstepniak 2 жыл бұрын
Thanks, Karla! Take care, and have a good day!
@Skyler9604
@Skyler9604 Жыл бұрын
My coworker suggested you for some Groovy vids and now I’m binging the rest of yours. Thanks for the great quality content, keep it up!
@szymonstepniak
@szymonstepniak Жыл бұрын
Oh man, I'm grateful for your binge watching session - it's the best thing a content creator can hear. Thank a lot! Take care, and have a good day!
@iceadobe139
@iceadobe139 3 жыл бұрын
Fantastic!! 👏🏼👏🏼 I learnt more in this video than I did in my two years of dev experience where I used JQ here and there by skimminging through the documentation examples for a particular use-case of mine.
@szymonstepniak
@szymonstepniak 3 жыл бұрын
Thank you so much for your kind words, Himalaya! It is very motivating to hear I was able to pack a lot of useful information to this short video :) Have a good day!
@diegochertoff
@diegochertoff 2 жыл бұрын
Very practical!! Thanks
@szymonstepniak
@szymonstepniak 2 жыл бұрын
Thanks for your kind words! Take care, and have a good day!
@teitoklien
@teitoklien 3 жыл бұрын
Loved your video ! , Good luck to your channel ! May it grow a ton
@szymonstepniak
@szymonstepniak 3 жыл бұрын
Thank you so much for your kind words, Soham! I hope it will start growing faster when I come back with the new videos in some near future :) Take care, and have a good day!
@teitoklien
@teitoklien 3 жыл бұрын
Thanks !, Yus I do believe it will grow , KZfaq’s a weird engine , one day you have a few thousand subs then suddenly you get into 50k+ , your content is a lot better than a ton of 200-300k+ programming youtuber’s I’ve shared some of your videos with my friends too , I’m confident you’ll be much more popular soon . Have a great day !
@CoronaBVW
@CoronaBVW 2 жыл бұрын
Dude, what I needed, real examples. I appreciate you taking the time to share.
@szymonstepniak
@szymonstepniak 2 жыл бұрын
Thanks for your kind support! Take care, and have a good day!
@DJAdams
@DJAdams 2 жыл бұрын
Loved this (and the others in the series), thank you. Nice production, good length, informative. And kudos for using values other than English in the sample data - that was a bonus to me.
@szymonstepniak
@szymonstepniak 2 жыл бұрын
Thanks for your kind words, DJ! I'm glad to hear you liked the exotic part of it :) Take care, and have a good day!
@RodrigoStuchi
@RodrigoStuchi 3 жыл бұрын
Woww, thanks for this jq series, really amazing!! 👏👏👏
@szymonstepniak
@szymonstepniak 3 жыл бұрын
Thanks for your kind words, Rodrigo! Glad you enjoy it! :-) Have a good day!
@iamfaze
@iamfaze 2 жыл бұрын
exactly what I was looking for. Thank you!
@szymonstepniak
@szymonstepniak 2 жыл бұрын
Thank you! Take care, and have a good day!
@deepambasu9661
@deepambasu9661 3 жыл бұрын
Great informative video for beginners. Awesome !!!
@szymonstepniak
@szymonstepniak 3 жыл бұрын
Thanks for your kind words, Deepam! Have a good day!
@pavelow36
@pavelow36 3 жыл бұрын
just amazing thank you
@szymonstepniak
@szymonstepniak 3 жыл бұрын
Glad you enjoyed it!
@sjcampbl
@sjcampbl 3 жыл бұрын
Really nice and informative series. Please continue. Thx
@szymonstepniak
@szymonstepniak 3 жыл бұрын
Thanks for your kind words, Shawn! Have a good day!
@user-ys4zg4ks7z
@user-ys4zg4ks7z 2 жыл бұрын
Я жду продолжения✌️
@ivanpavlov7101
@ivanpavlov7101 2 жыл бұрын
Very handy 😆 Thank you so much!
@szymonstepniak
@szymonstepniak 2 жыл бұрын
Glad to hear it was helpful for you, Ivan! Take care, and have a good day!
@pfkong6917
@pfkong6917 3 жыл бұрын
Amazing video you made! Learned a lot😃
@szymonstepniak
@szymonstepniak 3 жыл бұрын
Glad it was helpful! Have a good day!
@Maica1V
@Maica1V 2 жыл бұрын
Great video bro, I'm trying it out with kubernetes objects
@szymonstepniak
@szymonstepniak 2 жыл бұрын
Good luck! I hope jq does the job for you :) Take care, and have a good day!
@himanshutank5477
@himanshutank5477 2 жыл бұрын
I have always found your video helpful when I need to work with jq. I always refer them. I have a request to make if you can do a tutorial of using regular expression with jq. That would be awesome.
@szymonstepniak
@szymonstepniak 2 жыл бұрын
Thanks for your kind words and the suggestion 👍 I have to create some more jq related videos soon. Take care, and have a good day!
@XR5PL
@XR5PL 3 жыл бұрын
Cześć ,samo mięcho , dzięki !
@szymonstepniak
@szymonstepniak 3 жыл бұрын
Dzięki wielkie za ciepłe słowa :) Miłego dnia!
@tejeshvaish17
@tejeshvaish17 3 жыл бұрын
Awesome content man!
@szymonstepniak
@szymonstepniak 3 жыл бұрын
Thanks for your kind words, Tejesh! There will be more jq related videos published soon, so stay tuned!
@ronaldcastaneda2547
@ronaldcastaneda2547 4 жыл бұрын
Muchas Gracias, excelente explicación.
@szymonstepniak
@szymonstepniak 4 жыл бұрын
Thanks, Ronald! I'm glad you find it useful!
@michalginda387
@michalginda387 3 жыл бұрын
Great lessons!
@szymonstepniak
@szymonstepniak 3 жыл бұрын
Thank you for your kind words, Michal! Take care, and have a good day!
@saching9305
@saching9305 Жыл бұрын
Very thorough explanation of JQ! Great work🔥Could you please let me know, how did you screen-share and also your camera? Did you use Zoom or some other software to record yourself while screen sharing. Thanks in advance!
@szymonstepniak
@szymonstepniak Жыл бұрын
Thanks for your kind words, Sachin! My workflow is not the easiest one - I use a camera to record myself and I record screen separately using OBS. Then I edit it in Adobe Premiere. If you're looking for a tool that does both things at once, you might check Loom (or its alternatives.) I don't use it personally, but I've seen good results from other people and it looks like it is super easy to use. Good luck, and have a good day!
@saching9305
@saching9305 Жыл бұрын
@@szymonstepniak I really like the quality of your video production! Thank you so much for sharing your workflow, it is quite complex. Also, thank you for letting me know about loom, I'll check it out. You too have a great day!
@Gglsucksbigballz
@Gglsucksbigballz 2 жыл бұрын
I like your presentation. Not sure I like jq - I wanted to get away from a complex RegEx, and your (while easier to read) still seems like something I would forget and have to re-learn next time I come back to the code.
@j_holtslander
@j_holtslander 4 жыл бұрын
Enjoying your videos. Subscribed.
@szymonstepniak
@szymonstepniak 4 жыл бұрын
Welcome aboard, Jay, and thanks for the kind words! Have a good day! :)
@hsjoo1759
@hsjoo1759 2 жыл бұрын
is it okay to just add '| [limit(99999)] ' instead of wrapping whole thing with squre bracket?
@szymonstepniak
@szymonstepniak 2 жыл бұрын
If it works for you - feel free to do it. I tried replacing wrapping the whole expressions with the square brackets with the limit from your comment, but it didnt work for me. Take care, and have a good day!
@devashishkhandelwal1009
@devashishkhandelwal1009 2 жыл бұрын
Question: I have a json file which contains array of json. My objective is to parse that array of json and use each element as data in a curl req. For that i am writing a shell script and using "for" loop. but for some reason, output from jq parser for every iteration is not the json obj but each and every character within that json obj. For eg: if i try to parse "{"A":"B"}, the result would be : 1st iteration - {, 2nd iteration - "A:", 3rd iteration - "B", 4th iteration:}
@szymonstepniak
@szymonstepniak 2 жыл бұрын
Hi Devashish! I don't have out-of-the-box solution to your problem. It looks like your shell script uses a string to iterate using for-loop, which would explain the output you get. I would suggest using jq to extract a raw value from the JSON documents you want to pass to the curl command, so you end up with an output that contains each expected value separated by the new line, and then use for-loop to iterate over elements using a new line character as a separator (google something like "bash for loop new line"). Good luck!
@harishkolanu
@harishkolanu 3 жыл бұрын
how to get the values in one line like output should be title, author_name, ..
@mohammedminhajulhaq5542
@mohammedminhajulhaq5542 2 жыл бұрын
Hi Szymon, While trying to convert the filtered json output to array I am getting issues like all the data is coming in a separate array. My requirement is to get the count of a particular query. I saw your lesson and thought of using the length function but while trying to convert it into an array I am getting all the blanks array. Could you please help. Thanks for the wonderful lessons.
@szymonstepniak
@szymonstepniak Жыл бұрын
Hi Mohammed, do you have any example I could take a look on? You can use jqplay.org to share what the input JSON looks like and the code you are having problems with.
@thuyduong1809
@thuyduong1809 3 жыл бұрын
Thanks for your helpful video. Could you please tell me jq -r means?
@szymonstepniak
@szymonstepniak 3 жыл бұрын
Thanks for your kind words, Bui! The "-r" option stands for the "raw output". By default, jq filters serialize output with the JSON format, which means that e.g. every string is wrapped with double quotes. You can turn of this behavior with the "-r" option.
@thuyduong1809
@thuyduong1809 3 жыл бұрын
​@@szymonstepniak Merry Christmas! I appreciated your reply. I have the API endpoins below: curl -X GET "api.nsone.net/v1/zones" | jq -r ".[].zone" Outputs: hailee.tk haileeduong.net testduong.vn for zone in $(curl -X GET "api.nsone.net/v1/zones" | jq -r ".[].zone"); do curl -X GET "api.nsone.net/v1/stats/qps"; done Outputs: {"qps":52.0} {"qps":13.0} {"qps":7.0} But i want the outputs format like below: ZONE,QPS hailee.tk,{"qps":52.0} haileeduong.net,{"qps":13.0} testduong.vn,{"qps":7.0} So what should i add into the command? Love to hear advices.
@shreyash412
@shreyash412 3 жыл бұрын
which font do you use in the terminal? It looks good
@szymonstepniak
@szymonstepniak 3 жыл бұрын
The font I use is Source Code Pro Medium, size 12. Have a good day! :)
@vuha5889
@vuha5889 2 жыл бұрын
how did you do it can you share with me , thank you
@nitinsrivastava7458
@nitinsrivastava7458 3 жыл бұрын
Hi Szymon, I have a work requirement on JQ for which I can pay. I need your consultancy on the same. Please let me know if you are interested. My team is getting stuck with the task they are doing and they are involving PHP to do a certain set of tasks and then use JQ, I think JQ can alone do everything.
@szymonstepniak
@szymonstepniak 2 жыл бұрын
Hi Nitin! Thanks for your kind words. I don't take any consultancy gigs at the moment due to limited time. I hope you can find a good solution to your problem based on the available jq user guides and tutorials. Good luck, and have a good day!
@PaulLittlefield
@PaulLittlefield 3 жыл бұрын
Really well presented, thank you! Is it possible to merge the values of 2 fields and join them with a character but only if there is a certain field present? e.g. I only want to join the Series Name and Episode Name of the TV show as one value, but the Movie Name only has one value on its own... {"SeriesName":"The Adventures of Tintin","Name":"The Secret of the Unicorn (1)"} {"Name":"Guardians of the Galaxy"} ...becomes.. {"title":"The Adventures of Tintin / The Secret of the Unicorn (1)"} {"title":"Guardians of the Galaxy"} So far, I have this command line... jq -c '.[].NowPlayingItem | {SeriesName, Name} | select(.Name != null) | {title: (.SeriesName + " / " + .Name)}' emby_sessions.json ...which gives me the WRONG output (see the extra " / " ?) ... {"title":"The Adventures of Tintin / The Secret of the Unicorn (1)"} {"title":" / Guardians of the Galaxy"} Thanks :-)
@szymonstepniak
@szymonstepniak 3 жыл бұрын
Thanks for your kind words, Paul! Regarding your question, you can use if-then-else when you construct your final title field. In this case, you can replace: {title: (.SeriesName + " / " + .Name)} with something like this: {title: ((if .SeriesName then .SeriesName + " / " else "" end) + .Name)} and it should work as you expect. Hope it helps. Have a good day!
@PaulLittlefield
@PaulLittlefield 3 жыл бұрын
@@szymonstepniak omg, you star - absolute genius... it works perfectly! I have added it to the Emby Forum thread page and given you a thanks :-) emby.media/community/index.php?/topic/91982-username-nowplayingitemname-for-influxdb-and-grafana/&do=findComment&comment=952656
@szymonstepniak
@szymonstepniak 3 жыл бұрын
@Paul Littlefield Cowabunga, thanks! ☺️ I'm glad I was able to help you. Good luck and have a good day! 👍
Merging JSON documents in the command-line - jq tutorial
9:01
Szymon Stepniak
Рет қаралды 22 М.
Avoid These 5 Jenkins Pipeline Mistakes To Make Your Life Easier! 🔥
16:56
👨‍🔧📐
00:43
Kan Andrey
Рет қаралды 10 МЛН
ПОМОГЛА НАЗЫВАЕТСЯ😂
00:20
Chapitosiki
Рет қаралды 27 МЛН
Micronaut, GraalVM & Picocli - Build Native Java Command-Line Apps!
36:54
It's Been a Good Run, Drywall.
20:48
LRN2DIY
Рет қаралды 3,2 МЛН
JSON: Like a Boss
35:10
OICR Software Engineering Club
Рет қаралды 16 М.
Jenkins Pipeline for Java using Maven and Docker
17:43
Szymon Stepniak
Рет қаралды 29 М.
Linux File System/Structure Explained!
15:59
DorianDotSlash
Рет қаралды 4,1 МЛН
Jenkins Pipeline Tutorial: Declarative Pipeline vs Scripted Pipeline
12:41
EVERYONE Needs to Learn a Little Bit of AWK!
16:29
Gary Explains
Рет қаралды 396 М.
👨‍🔧📐
00:43
Kan Andrey
Рет қаралды 10 МЛН