I’ve started this series of posts about general programming in whatever language I happen to feel like. I decided to make it a series of posts separate from from my game development dev log series of posts. This one is focusing on some ideas I have for a couple of different utilities in Python specifically.
Since entry 2 I did spin off this program to a separate repository on my GitHub. I also realized just how much of a mess entry 2 really is (sorry). For this one I decided I’d like to cut off at a natural end point.
All of these Programming Journey posts can be found in the associated category of this blog.
Lets get more specific
I didn’t work on this very much this week but when I finally did start working on it again I realized I really need to think even more about what I actually want this to be. I can’t start on the foundation until I really nail down what I want the final thing to look like.
Well the most obvious summary would be to bring in my list of purchases from the Steam store and present them with a list of additional options not provided on the Steam profile page version. With filtering and adding my own metadata and writing that back in to…a database for lack of a better term. And perhaps some conversions between formats. Like maybe after bringing in the data to read, modify and add to I can then export back out to CSV or whatever other format. That’s about as much of a summary as I can come up with: bring in the raw data from the Steam profile page, convert the data to JSON format so that it can be used in Python as a dictionary or python object etc and then have a way to write those changes back out to one or more formats to have my local copy essentially.
So if that summary were broken down into three very over-simplified parts:
- Bring down raw data from steam profile page
- Write extracted data out to file for easy viewing and editing
- Ability to export edited data back out to a file in any number of standard formats
I think I’m almost done with the first point. Although that second point is going to have I imagine ten or fifteen steps and questions to answer all to itself. Like how do I want to present it? What does the UI look like for editing data? What are some of the methods for extracting metadata? I don’t really have answers tot these questions. I’m kind of winging it as I learn more.
Luckily I don’t need to worry about step two until I definitively have step one down. And on that I have made some progress.
First I didn’t know if the program should start by asking for the steam ID in question or not. If I asked for the steam ID I could then concatenate the first and second half of the URL together with the ID for the global variable. As in SteamID = Tildes then firstHalfURL = ”’https://steamcommunity.com/id/”’ and SecondHalfURL = ”’/games/?tab=all”’ then tied together with SteamURL = firstHalfURL + SteamID + SecondHalfURL. That Would theoretically tie them all together into one string.
I might come back and insert a prompt later but for development and debugging purposes having to paste in an ID every time I run the script would get old really quick. So I commented out that could and left in my hypothetical constant/global from the prior entry.
The next most obvious task was some way just get the data I wanted and write it to a text file so I don’t have to hit the Steam URL every time I run the script. Unless a user were constantly adding/deleting things from a library that seems unnecessary.
Several questions crop up just from that:
- How do I determine the file name for the game data?
- I think I’ll use the steam ID in the file name
- How do I determine if the file name already exists?
- How does the program react based on the file existing?
For the first question about the file name the most obvious answer would be to prompt the user for a file name (and location to save the file for that matter).
But for development purposes, I decided instead to use the steam ID in the file name.
# Technically I should concatenate together the URL as well but this is fine for now
STEAM_LIBRARY_URL = '''https://steamcommunity.com/id/nelixery/games/?tab=all'''
STEAM_USER_ID = '''nelixery'''
SAVED_DATA_FILE_NAME = 'gamedata-for-' + STEAM_USER_ID + '.txt'
I should mention that I have no idea who nelixery is besides being a steam user that has only a eight steam purchases listed on his public profile page. I just chose the profile because eight entries is much easier to work with than my own library with 1,000+ entries.
I went through a few different iterations for question two, but ultimately ended up with this function that ends with returning either True or False:
# required module for filesystem related functions/tasks like "exists"
# is this pathlib
from pathlib import Path
def CheckFileExists():
p = Path(SAVED_DATA_FILE_NAME)
if (p.is_file() and p.exists()):
#print("File exists and is a file")
return True
else:
#print("File does not exist")
return False
Well this is a start on my main task of not hitting Steam every time the script runs.
I also have this block that uses my newly minted CheckFileExists function:
# pretty straight forward: Does the file exist?
# if not write the file, if it does already skip to next
# something a place holder for now
if CheckFileExists() == False:
with open(SAVED_DATA_FILE_NAME, "w+") as file:
file.write(justGamedata)
file.close()
print("file written")
else:
print("file exists")
This block of code is missing a few things. Like exception handling for instance, or actually asking the user for input on the process.
The last most obvious step would trying to combine the first part of “does the file exist” with the “also if it does exist no need to GET the data again” via the REQUESTS module.
There’s a chance I made this overly complicated. But I wrote two more functions: one just for retrieving the steam data and to determine if a file needs to be written or not.
def RetrieveSteamData():
# use get to grab the HTML source
result = requests.get(STEAM_LIBRARY_URL) # resut.text will show whole source of page
# make this into HTML
loadToBeaut = BeautifulSoup(result.text, "html.parser")
# select "only" the appropriate <script> tag with the JSON data
scriptContent = str(loadToBeaut.select('#responsive_page_template_content > script:nth-child(4)')[0])
# jump to first character of JSON string (position)
#PostambleStartPos = (scriptContent.find("= ") + 2)
PostambleStartPos = (scriptContent.find("= ") + 3)
# jump to last character of JSON string (position)
#startSuffixPos = scriptContent.find("rgChangingGames") - 8
startSuffixPos = scriptContent.find("rgChangingGames") - 10
justGamedata = scriptContent[PostambleStartPos:startSuffixPos]
return justGamedata
This is the previously posted code except a little shorter and it returns the string that is the data.
So next is a bring-it-all together function that calls both RetrieveSteamData() and CheckFileExists().
def WriteOrOpenDatafile():
# determine if file already exists
if CheckFileExists() == False:
# if not then assign a variable to the string value returned from
# RetrieveSteamData()
DataFile = RetrieveSteamData()
with open(SAVED_DATA_FILE_NAME, "w+") as file:
file.write(DataFile) # write said variable to a file
file.close() # don't forget to close file out
print("file written") # more a debugging thing
else:
# if this file already existed then nothing to do
print("file exists")
# well really the else should open the file for reading
# Next final step is to actually call the bring-it-all-together function
WriteOrOpenDatafile()
I put these functions together without an existing file to start: the first time I ran the program, it wrote the file while the second time I ran the program (the file now existing) it printed “file exists” as expected.
Now it actually feels like like I’m making some progress. I’ve decided to end this entry here and continue in the next entry. Don’t want a repeat of entry 2.
Reference links:
- Automate the Boring Stuff (free book via web site)
- Automate the Boring Stuff 15 part YouTube playlist (from 2015 but still good for basics)
- JSON via Requests lib YouTube video (mCoding)
- Reference: write a new file in python (W3Schools)
- A page with a better explanation of curses I found
- Python and JSON via “Geeks for Geeks”
- An all-in-one reference page for Python (Learn X in Y Minutes)
I've made a "clearing house" repo for random python programming projects on GitHub: https://github.com/tildesarecool/ReallyHadtoPython
Tildes on the side 
One thought on “Programming Journey Entry 3: We’re all JSON now”