|
Project Information
|
StatusAppReduce code is not yet released. We plan to release the the code along with the below design doc, so please check back soon! ObjectiveAppReduce primarily aims to reduce commercial application license spending. It does this by making license costs more transparent to employees and by providing them with self-service, automatic mechanisms for uninstalling applications. In addition to its primary goals, AppReduce can help an organization meet various secondary goals such as encouraging the use of free applications, relying on cloud-based solutions, simplifying the installed IT footprint, providing a web-based interface to view licenses, etc. BackgroundSurely at some point everyone has installed a commercial application, used it a few times, then had it sit idle on their computer. Often such applications have associated licensing costs and therefore money has been spent on something that is not being used; said licenses can cost as little as $5 and as much as $1000 or more. With many licensing agreements customers pay a true-up fee at a fixed interval, meaning they pay based on the number of instances across an organization regardless of which specific employees or computers have the application. In these cases, if a user uninstalls a piece of previously licensed software it typically does not equate to a refund from the software manufacturer; however the next time a different user needs that same application and installs it, a new license does not need to be purchased. The license paid for originally becomes available for reuse because the original user has uninstalled the application. Furthermore, some manufacturers allow their customers to pay a premium that entitles them to free upgrades to the newest software versions. For example, if such an agreement exists with CompanyX and a user upgrades CompanyX ApplicationY version 2009 to ApplicationY version 2010, a new license would not need to be purchased. In some cases this may also mean one employee could uninstall CompanyX ApplicationY 2009 and a different employee could install CompanyX ApplicationY 2010, and the old 2009 license could be used instead of purchasing a new license for 2010. Employees may legitimately need commercial software, even when it is used infrequently. However, almost all employees can benefit from an easily accessible reminder of their installed licensed software, whether or not they choose to uninstall any of that software. Non-Goals
OverviewUpon visiting the AppReduce site an employee see a list of licensed software that is installed on their computer(s), along with the associated licensing costs of these applications. The employee can automatically uninstall application(s) with the click of a button. Statistics regarding cost per user, per group and per overall organization are displayed to provide users with a relative perspective on their license costs. InfrastructureAppReduce uses the following infrastructure and services:
Design Chart
General Website Flow
Data Models
def AddUninstallToAlltimeStats(computer, application):
"""Adds application uninstall to the stats cache entities.
Args:
computer: a Computer object.
application: an Application object.
"""
# Add cost to total_savings.
total_savings = models.KeyValueCache.get_or_insert(
'total_savings', float_value=0.0)
total_savings.float_value += application.cost
total_savings.put()
# Add 1 to total_uninstalls count.
total_uninstalls = models.KeyValueCache.get_or_insert(
'total_uninstalls', int_value=0)
total_uninstalls.int_value += 1
total_uninstalls.put()
# Add 1 to unique users, but only if user hasn't uninstalled before.
user_is_unique = models.UninstallLogEntry.all().filter(
'owner_uid =', computer.owner.uid).get() is None
if user_is_unique:
total_unique_user_uninstalls = models.KeyValueCache.get_or_insert(
'total_unique_user_uninstalls', int_value=0)
total_unique_user_uninstalls.int_value += 1
total_unique_user_uninstalls.put()Data Collection / BulkloadA Python script running on a regular basis (cron) fetches computer and employee information from Corporate Inventory Systems to import to AppReduce. The following models are filled on each bulkload execution: Computer, Googler, ApplicationCost, ApplicationCostHistory, GroupedComputerCost, and GroupedComputerCostHistory models. The following steps are performed:
App Engine TaskQueue UsageGAE TaskQueues are used to perform background jobs. AppReduce uses TaskQueues for many operations that generally all follow the same model: do as much as possible in the given App Engine request time limit, catch the DeadlineExceededError exception, defer the remainder of the job to a new Task (a new request). Example pseudo code: def Foo():
try:
... code which may take longer than the request limit to execute ...
except runtime.DeadlineExceededError:
logging.info('Request limit hit. Deferring Foo to future Task...')
deferred.defer(Foo, _name='foo-task-name')
return
logging.info('Foo fully completed!')A more specific example of how this is done in AppReduce can be seen in the data import model cleanup. Because the data is authoritative in Corporate Inventory Systems, not AppReduce, entities that didn't get imported after each import cycle need to be deleted. This determination is based on the assumption that the reason these entities were not imported is that they've likely been deleted or disabled in the authoritative data source. AppReduce handles this deletion as follows: the mtime value is shared across all entities updated in a given data import cycle, so after the cycle it's safe to delete any entities where mtime is less than the newest mtime in the Kind. See the TruncateModelByMtime code below: import datetime
import logging
import models
from google.appengine import runtime
from google.appengine.ext import db
from google.appengine.ext import deferred
BATCH_DELETE_SIZE = 200
def TruncateModelByMtime(model_name=None):
"""Truncates a given model for all entries older than the latest mtime.
Note: This only works on models with an "mtime" field. Furthermore, there is
no padding or horizon time, so if any mtime is even a second before the
latest then it gets deleted. When bulkloading models to be used with this
function, ensure all entities share a common mtime.
Args:
model_name: str model name to truncate.
"""
if model_name is None:
logging.error('Truncate Model: model is None')
return
elif model_name.endswith('History'):
logging.error('Truncate Model: History tables is not allowed.')
return
elif not hasattr(models, model_name):
logging.error('Truncate Model: model does not exist %s', model_name)
return
model = getattr(models, model_name)
if not hasattr(model, 'mtime'):
logging.error(
'Truncate Model: model "%s" does not have field "mtime".', model_name)
return
logging.info('Truncate Model: Truncating %s', model_name)
# Get a single entity with the latest mtime from the model.
desc_mtime_ent = db.Query(model).order('-mtime').get()
latest_mtime = desc_mtime_ent.mtime
logging.info('Deleting entities with mtime < %s', latest_mtime)
# Get the keys of all entities with mtime < latest mtime.
keys = db.Query(model, keys_only=True).filter('mtime <', latest_mtime)
num_deleted = 0
try:
to_delete = []
for key in keys:
to_delete.append(key)
if len(to_delete) == BATCH_DELETE_SIZE:
db.delete(to_delete)
num_deleted += BATCH_DELETE_SIZE
to_delete = []
# Delete remaining keys; if to_delete didn't reach batch size in last loop.
if to_delete:
db.delete(to_delete)
num_deleted += len(to_delete)
except runtime.DeadlineExceededError:
# Run this function again in a new TaskQueue entry in case more rows exist.
logging.info('Deleted %d entries. Deferring...', num_deleted)
now_str = datetime.datetime.utcnow().strftime('%Y-%m-%d-%H-%M-%S')
name = 'truncate-model-%s-%s' % (model_name, now_str)
deferred.defer(
TruncateModelByMtime, model_name=model_name, _name=name)
return
logging.info('Deleted %d entries. Complete!', num_deleted)App Engine Cron UsageGAE Crons are used (typically with TaskQueue/Deferred) to do the following:
ScalabilityVirtually all read operations are cached in our App Engine datastore, and many in memcache as well. GORD is only hit via SDC for write operations, and on-demand data refresh requests. AppReduce can handle a very high load in the context of an internal/corporate application. During the Google-internal launch the application sustained 20 QPS for an extended period of time. Below is a code snippet to fetch grouped data from the Datastore, which is notable for two reasons: 1) it's fairly expensive and therefore memcache is used, 2) In Datastore != (not equal) queries are slower than Python looping in certain circumstances, as != queries are actually two separate queries under the hood. def GetMainReportsData():
"""Gets data needed to load the main reports page.
Returns:
Tuple. Sorted lists of departments, cost centers, applications, and quarters
where uninstalls have occured.
"""
main_report_data = memcache.get('main_report_data')
if main_report_data is not None:
return main_report_data
all_departments = models.GroupedComputerCost.all().filter(
'type = ', 'department').order('name')
all_cost_centers = models.GroupedComputerCost.all().filter(
'type = ', 'cost_center').order('-cost').fetch(10) # Only top 10.
all_applications = models.ApplicationCost.all().filter(
'group_type = ', 'total').order('name')
# Doing a DataStore != query is less efficient than doing here.
departments = [department.name for department in all_departments
if department.name != 'Unknown']
# Doing a DataStore != query is less efficient than doing here.
cost_centers = [cost_center.name for cost_center in all_cost_centers
if cost_center.name != 'Unknown']
applications = [application.name for application in all_applications]
# Generate years/quarters for the RoI reports.
uninstalls = common.GetUninstallsForPastDays()
quarters = set()
for u in uninstalls:
quarter = misc.GetQuarter(u.time_requested.month)
quarter_string = '%d Q%d' % (u.time_requested.year, quarter)
quarters.add(quarter_string)
data = (departments, cost_centers, applications, sorted(list(quarters)))
memcache.set('main_report_data', data, common.DEFAULT_MEMCACHE_SECS)
return dataWork EstimatesEstimated timeline:
During and after rollout the service needs to be monitored for usage, compared to goals, and tweaked with new marketing ideas/plans for better coverage. Due to this, an unknown but fairly low amount of work may be needed past launch. As new applications and versions are released, new unattended uninstallations also need to be created and added to AppReduce; however, this should be fairly rare/infrequent. CaveatsA simple dashboard showing application license spend without the automated uninstall packages is not ideal for a number of reasons:
AppReduce highly depends on the data integrity of SCCM. There are a number of scenarios where data may be inaccurate:
Microsoft, Windows, System Center Configuration Manager, System Management Server, SQL Server are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. Mac either registered trademarks or trademarks of Apple Inc. in the United States and/or other countries. |
|
