iHeinrich

just trying to find the answer…

Category: Ruby

Fresh Fedora Install? Do this next…

A while back, I installed Fedora 27 on a Dell laptop. I installed it as a partition, so I could dual boot either Windows 10 or Fedora. After a while, I found myself using only Fedora. So I reformatted the drive (using LVM whole disk encryption) and now it’s a Fedora only machine. If I want Windows, I’ll create a VM image.

Update your install

I know you think, “I just installed this, it must be updated…” Yeah, well…. no.

Do this

sudo dnf upgrade --refresh

The instructions are detailed here https://fedoraproject.org/wiki/DNF_system_upgrade.

The upgrade will take a while but it’s worth it. After it’s done, a system reboot is a good idea (it couldn’t hurt).

Install GCC and Ruby

Bizarrely, GCC and Ruby are not installed by default. But you can check by typing:

gcc --version

and if you don’t have gcc installed, you will be offered the choice to install it.

bash: gcc: command not found...
Install package 'gcc' to provide command 'gcc'? [N/y]

Say yes.

Now for Ruby.

ruby -v
bash: ruby: command not found...
Install package 'rubypick' to provide command 'ruby'? [N/y]

again, say yes.

That’s all for this installation of “Fun with Fedora”.

 

 

Have you read this PDF?

Google-PDF-Image-Result-150x150A friend of mine came to me with interesting situation.

He was a teacher at a school where the student’s email address was their student number + the school server. If you were student number 121, your e-mail address was 121@theSchool.edu.

You get the idea.

The report cards were in PDF form, and they wanted to mail off the report cards to the students. How were they planning on doing this?

  1. Gather a group of teachers in a conference room with their computers
  2. Give them a thumb drive full of report cards on PDFs
  3. Have the teacher open the PDF file
  4. COPY the student number
  5. Create an email, using the COPIED student number (PASTE and type “@theSchool.edu”)
  6. Copy from ANOTHER text source the message, “Dear Student, yakkity yak yak. Here is your report card. Read ’em weep. Love, the School”
  7. SEND email
  8. Rinse, lather and repeat

wearing a mask, a cape and armed with mad Ruby skillz, you can save the world (or at least a roomful of teachers)

This looks like a job for RUBY SUPERHERO!

Part One: Reading PDFs.

Like most tough things in Ruby, there is a GEM for that. In the case of parsing PDFs, you need to install the PDF Reader Gem which is located at:

https://github.com/yob/pdf-reader

Using this Gem, we can read the PDF and put it into a string format and then extract the data we need.

After installing the GEM, your script will start with

#!/usr/bin/env ruby
require 'pdf-reader'

Part Two: Counting the (Report) Cards

Since we have a folder full of PDFs, we need to read the folder and get a count of the PDF files in the folder.


theCount = Dir.glob('*.pdf').count
puts "there are " + Dir.glob('*.pdf').count + " files to e-mail"

I like to toss in friendly put messages so when things go horribly wrong, you get some indication of where it broke.

Part Three: Iterate thru the folder


Dir.glob('*.pdf') do |rb_pdf|

# lots of Ruby Magic! 

end

This nifty line says, “Read the directory, looking ONLY at files that end in .pdf and call the elements “rb_pdf”. As you go thru the elements one by one, do stuff to them.

The stuff part is coming.

Part Four: Read the file!


reader = PDF::Reader.new(rb_pdf)

This is where the GEM pdf-reader comes in. It will read the file element “rb_pdf” and creates an object called “reader”.

Object “reader” has a number of elements, but we are most interested in the text of the object.


longString = page.text

Now that the ENTIRE report card is a string, we just need to find the student number in that string data so we can generate the email address.

But wait, how to find the number in that haystack of data?

Part Five: Oh yeah, it’s REGEX time!

Regular Expressions, or regex, is a great way to find most anything in an ocean of data, but it has a brutal learning curve and it’s very unforgiving to newbies. There are lots of great resources and tools online to help you with Regex, I suggest you use them.

I know the student number is the only 6 digit number in the PDF so I’ll look for that.


studentNumber = longString[/\b\d{6}\b/] 
#returns the six digit number for emailing
eMailTarget = studentNumber + '@theSchool.edu'

Part Six: Mailing ain’t easy
So, you test the code and it all works great… but how to mail?

This was the hardest part, and your milage may vary depending on the mail server configuration where you are.

You might need to install the mail Gem, depending on where you are.


require 'mail'  # ruby mail library. https://github.com/mikel/mail
require 'openssl' #sometimes, Outlook just makes you crazy...
#Sending via Outlook

    Mail.defaults do
      delivery_method :smtp, { 
                               :address              => 'mail.theSchool.edu',
                               :port                 => 587,
                               :domain               => 'theSchool.edu',
                               :user_name            => 'theSchool/poorTeacher',
                               :password             => 'summerVacation',
                               :authentication       => :login,
                                :enable_starttls_auto => true,
                                :openssl_verify_mode => OpenSSL::SSL::VERIFY_NONE  
                               }
 
    end
                          
    # send test message
    Mail.deliver do
 
        from    'poorTeacher@sts.theSchool.edu'
        to      eMailTarget
        subject 'Report Card'
        body    'Congratulations on getting a report card'
        add_file :filename =>  rb_pdf
    end
    
    puts "mailed to " + eMailTarget  
    # end of mailer part

Summary
So what have we learned?

  • If you are doing the same thing 10 or 20 times over, it means a script should be doing it.
  • You can read pdfs using a Ruby Gem.
  • Regex is wicked powerful and can be wicked hard to figure out.
  • Outlook can drive you crazy if you are trying to automate something.

Updating MacOS X to Ruby 2.0

ruby2MacOS X comes with ruby by default. It’s great, but what if you want to join the cutting edge and use Ruby 2.0 and Rails 4 like all the cool kids?

First, let’s look at which version we have installed.


Heinrichs-iMac:~ hbeck$ ruby -v
ruby 1.8.7 (2012-02-08 patchlevel 358) [universal-darwin12.0]

I like things simple, so let’s find a simple way to update this.

Homebrew

Fortunately there is a simple package manager called “Homebrew” which makes life very easy. I like easy.

Installing it takes one line in the terminal:


Heinrichs-iMac:~ hbeck$ ruby -e "$(curl -fsSL https://raw.github.com/mxcl/homebrew/go)"

After you install, pay attention to the instructions. You may need to run a command to update it, which you should do.

rbenv

Some people prefer to manage their versions of Ruby using RVM. I prefer to use rbenv. It’s simpler and lightweight. The documentation for rbenv is here.


Heinrichs-iMac:~ hbeck$ 
brew update
brew install rbenv
brew install ruby-build
brew install openssl  #this part might be optional, if it is, you will get a notice saying, "already installed"
CONFIGURE_OPTS=--with-openssl-dir=`brew --prefix openssl` rbenv install 2.0.0-preview1

the last command will take a while to run.

When it finishes, you can run this command to see WHICH versions of ruby are installed


Heinrichs-iMac:~ hbeck$ rbenv versions
* system (set by /Users/hbeck/.rbenv/version)
  2.0.0-preview1

So the asterisk shows that there is a system ruby (which we already is know 1.8.7) and a cool hotrod 2.0.0-preview1 available.

Since we installed rbenv, we need to tell our $PATH profile to use the settings from rbenv and then “restart” the shell to use the new settings


Heinrichs-iMac:~ hbeck$ echo 'eval "$(rbenv init -)"' >> ~/.bash_profile
Heinrichs-iMac:~ hbeck$ exec $SHELL -l

Let’s make that hotrod switch! You will need to CAREFULLY type the name of the version, so watch that you get all the characters correct.


Heinrichs-iMac:~ hbeck$ rbenv local 2.0.0-preview1

So let’s verify that we have the correct version.


Heinrichs-iMac:~ hbeck$ ruby -v
ruby 2.0.0dev (2012-11-01 trunk 37411) [x86_64-darwin12.4.0]

I’ve been duped!


dupedIn testing my Photo Organizing Script, I noticed something scary.

If I move an image into a folder where another DIFFERENT image has the SAME name, the move function overwrites previous image.

No warning.

This is a pretty serious issue, since it will result in data loss.

It’s time to write a function to detect for this and rename the file.



	def move_safely (file, folderName)

	  if File.exists?(folderName+"/"+file)
	    puts "the file exists and should not be overwritten!"

	    #rename the file
	    #generate an 8 char random string to append to the file name.
	    random8 = (0...8).map{(65+rand(26)).chr}.join

	    #break up the file into file and file type. The result of this is an array
	    breakUp = file.split(".")

	    #add the fixin's to the random8 string
	    randomize = "_"+random8+"."

	    #insert it into the array
	    breakUp.insert(1,randomize)

	    #convert the array into a file string
	    newFileName = breakUp.join

	    File.rename(file,newFileName)

	    file = newFileName

	  else
	    puts "the file doesn't exist"
	  end

	FileUtils.move file, folderName

	end

NEF images with Sorting Script


logo-NEF
I was pretty happy with my scanning and sorting script.

I shoot with a Nikon camera, so many of my images are NEF/RAW images.

Unfortunately, the exifr method JPEG.new doesn’t like NEF images, because they aren’t JPEGS.

Maybe they are TIFFs? Let’s give it a try!



if file.end_with?(".NEF")
     dateShot = EXIFR::TIFF.new("#{file}").date_time
     puts "it's a NEF"
else
     dateShot = EXIFR::JPEG.new("#{file}").date_time
end

so we replace the one line that extracts date with this “NEF checker” and it works!

Drowning in a sea of pictures…..


Great_Wave_off_Kanagawa2If you are like me, you take a gazillion pictures and they are all over the place.

Recently, I decided to the consolidate all my data from all my hard drives onto one massive NAS Drive (Synology 412+).

Synology DOES offer a tool that let’s you sort all your images using “Smart Folders” or something, but that didn’t appeal to me.

I wanted a ruby script to that I could drop into a sea of pictures and I would get all the images sorted by month and year into named folders.

I am sure there are tools that do this, but I wanted to write my own.

I’m stubborn that way.

Step 1: Install exifr reader gem.

One of the great things about ruby is that there usually is a “gem for that”.


Heinrichs-iMac:~ hbeck$ sudo gem install exifr

WARNING: Improper use of the sudo command could lead to data loss
or the deletion of important system files. Please double-check your
typing when using sudo. Type "man sudo" for more information.

To proceed, enter your password, or type Ctrl-C to abort.

Password: StevePontEsMuyMacho
Successfully installed exifr-1.1.3
1 gem installed
Installing ri documentation for exifr-1.1.3...
Installing RDoc documentation for exifr-1.1.3...
Heinrichs-iMac:~ hbeck$ 

Step 2: Write some code!

We are going to require

  • rubygems since we are using a gem.
  • exifr, to read the exif data.
  • date, to read the date.
  • fileutils, to move the files.

#! /usr/bin/env ruby

require 'rubygems'
require 'exifr'
require 'date'
require 'fileutils'


# I want to just drop this in a folder of images. This gets the working directory
wd = Dir.getwd

# files is an array of file name strings in the working folder.
files = Dir.entries("#{wd}")

#I'm going to go thru each file and read it's date.
files.each do |file|  
        # the meat of what I want to do will go here....
end

Step 3: There’s trouble right here in Directory City

One of the pain in the keister issues in MacOS is that hidden files called “.” or “..” will be picked up in the directory scan. But since they do not have exif data we need to sniff for them and NOT run the exifr method on them. Also, the ruby file will be in the directory so I need to exclude that as well.


if file.start_with?(".") || file.end_with?(".rb") 
    puts "(hand wave) this is not the file you are looking for..."
  else
       # the meat of what I want to do will go here.
  end 

Time to get some shooting data!


dateShot = EXIFR::JPEG.new("#{file}").date_time

returns a date that looks like this: Sat Jun 22 22:09:22 -0400 2013

I know it’s a date class since I double checked thusly:


puts dateShot.class

and got ‘Date’.
If ever you are unsure what a variable actually is ‘variable.class‘ is your best friend.

Step 4: There’s MORE trouble right here in Directory City

The idea is to make directories with Month_Year, but what if there is ALREADY a directory in the folder? What if I dump a sea of images into a directory, run my magic ruby script, and then find MORE images after I have sorted all the images into directories? I can’t run the exifr method on a directory, so I need to sniff for that as well.


if file.start_with?(".") || file.end_with?(".rb")
    puts "(hand wave) this is not the file you are looking for..."
  elsif file.include? "."
     # I know that all my jpeg files will have a DOT in the name and my directories do NOT have dots in their name.
     # So I will put the methods here....
  else
    puts "it's a directory"
  end 

Step 4: How about a date, baby?

I want the date formatted like this: Mon_Year. The Date class offers a method to format our date pretty much any way we like.


     dateShot = EXIFR::JPEG.new("#{file}").date_time
     folderName = dateShot.strftime("%b_%Y")

This will return Jun_2013.

I want to make the directory, but of course, if the directory ALREADY EXISTS, I want to leave it alone.


 if File.exists?("#{folderName}") # I thought Dir.exists would work, but it doesn't. If you know WHY, please let me know.
          puts "the directory exists"
     else
          Dir.mkdir("#{folderName}")
          puts "make the directory"
     end

The directory has been created, let’s move the files in there…


FileUtils.move file, folderName

Yes, it really is that easy.

Step 5: It’s ALWAYS SOMETHING ALICE!! 

Ok, you are thrilled that everything is working fine and images are getting sorted out but then exifr finds that SOME images have no shooting data.

It happens. Maybe you downloaded the image from a website that stripped out the exif data.

When this occurs, the exifr method returns nil.

We need to catch that, so let’s make a NO_Date directory and drop the undated images in there.


   if dateShot.nil?
         folderName = 'No_Date'
         puts "there is no date"
 else
         folderName = dateShot.strftime("%b_%Y")
 end

Step 6: Let’s test it! 

I suggest you start with a small folder of images, drop the ruby file in there, and see your results. I wrote this on my MacOS, but it should work on any OS. If you have Windows, it goes without saying that you need to install Ruby to make this work (but I said it anyway).

Step 7: What happens if I have a NEF/RAW image? 

Good question! I’ll investigate this another time. I have a lot of images to clean up!

Here is the final script. Enjoy.


	#! /usr/bin/env ruby

	require 'rubygems'
	require 'exifr'
	require 'date'
	require 'fileutils'

	# I want to just drop this in a folder of images. 
	wd = Dir.getwd

	# files is a list of a files in the working folder
	files = Dir.entries("#{wd}")

	#I'm going to go thru each file and read it's date.
	files.each do |file|  

	  if file.start_with?(".") || file.end_with?(".rb") 
		#ignore ruby scrip and hidden files.
	    puts "(hand wave) this is not the file you are looking for..." 
	  elsif file.include? "."
	     dateShot = EXIFR::JPEG.new("#{file}").date_time

	     if dateShot.nil?
	       folderName = 'No_Date'
	       puts "there is no date"
	     else
	       folderName = dateShot.strftime("%b_%Y")
	     end

	     if File.exists?("#{folderName}") 
	     	puts "the directory exists"
	     else
	     	Dir.mkdir("#{folderName}")
	     end

	     FileUtils.move file, folderName

	  else
	    puts "it's a directory"
	  end

Final thoughts

Happy Ticket


While on the bus in Russia, they give you a ticket with a 6 digit number.

The Russian I was with closely read the numbers, and I asked “Why do you care about the number?”

She said, “If the first three numbers equal the last three numbers, it’s a ‘Happy Ticket’ and I keep it.”

I tried to figure the odds on getting a “Happy Ticket” and just gave up.

I’m a programmer, not a mathematician.

So I wrote a Ruby script to iterate thru the numbers and do a comparison of the first three and the last three.


#!/usr/bin/env ruby
i = 0
happy = 0
theCount = Array.new
while i < 999999
i += 1
f = sprintf '%06d', i # f is a six digit number

theCount = f.chars.to_a

first = theCount.fetch(0).to_i + theCount.fetch(1).to_i + theCount.fetch(2).to_i
last = theCount.fetch(3).to_i + theCount.fetch(4).to_i + theCount.fetch(5).to_i

     if first == last
          happy += 1
          print("Happy Number ", happy," is ", theCount[0..2], theCount[3..5], "\n") 
      end

end

The end result is that there are 55,252 'Happy Numbers', meaning your odds of getting one is about 5.5%

Powered by WordPress & Theme by Anders Norén