Fedora Linux Support Community & Resources Center
  #1  
Old 28th April 2017, 04:17 PM
rexrf Offline
Registered User
 
Join Date: Aug 2016
Location: Dallas
Posts: 47
linuxfedorachrome
file manipulation.

Hey Everyone,

I have a script that collects packet statistics that im trying to add a new feature to, but cant quite figure something out. I do a a curl and receive information in JSON. Which looks something like:

Code:
{
  "ip": "178.236.4.190",
  "hostname": "No Hostname",
  "city": "Dublin",
  "region": "Leinster",
  "country": "IE",
  "loc": "53.3389,-6.2595",
  "org": "AS16509 Amazon.com, Inc."
}
Then I use a tool, jq, to break the info down into smaller chunks. The problem is that when i try to uniq and sort the data i am still getting duplicate lines. For instance:

Code:
   124 "Virginia"
    119 "California"
    113 "Washington"
    112 "Virginia"
    100 "New Jersey"
     96 "Washington"
     92 "California"
I've tried removing the quotes, converting the characters to all lowercase, and removing empty lines, none of which seems to work for me.

any Ideas?


Thank Much,
Rex
Reply With Quote
  #2  
Old 28th April 2017, 04:28 PM
Jeff Sadowski Offline
Registered User
 
Join Date: Jun 2005
Age: 43
Posts: 494
linuxchrome
Re: file manipulation.

Sort must happen before uniq. There was an article on lxer.com a couple weeks back using awk for both as it is faster and understands unicode.
Found the article that was linked from lxer.com
http://www.thelinuxrain.com/articles...disappointment
Reply With Quote
  #3  
Old 28th April 2017, 04:40 PM
rexrf Offline
Registered User
 
Join Date: Aug 2016
Location: Dallas
Posts: 47
linuxfedorachrome
Re: file manipulation.

Quote:
Originally Posted by Jeff Sadowski View Post
Sort must happen before uniq. There was an article on lxer.com a couple weeks back using awk for both as it is faster and understands unicode.
Found the article that was linked from lxer.com
http://www.thelinuxrain.com/articles...disappointment
Well that fixed it, here the exact pipe, if anyone runs into a similar issue:
Code:
sort foo.txt | uniq -c | sort -nrk1 > bar.txt
Thanks again!
Reply With Quote
Reply

Tags
file, json, manipulation, sort, uniq

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
mbr manipulation paulywauly Hardware & Laptops 14 2nd May 2012 05:23 PM
Text file manipulation with bash tutorial BeerGut Using Fedora 11 24th October 2009 06:37 AM
file manipulation madjac Using Fedora 1 13th December 2006 12:16 AM
LVM manipulation fromans4 Using Fedora 13 15th October 2006 10:18 AM


Current GMT-time: 21:43 (Wednesday, 20-09-2017)

TopSubscribe to XML RSS for all Threads in all ForumsFedoraForumDotOrg Archive
logo

All trademarks, and forum posts in this site are property of their respective owner(s).
FedoraForum.org is privately owned and is not directly sponsored by the Fedora Project or Red Hat, Inc.

Privacy Policy | Term of Use | Posting Guidelines | Archive | Contact Us | Founding Members

Powered by vBulletin® Copyright ©2000 - 2012, vBulletin Solutions, Inc.

FedoraForum is Powered by RedHat