PySpark User Activity Analysis on Cloudera VM
A skill to join user activity and user info CSV datasets using PySpark 1.6 on Cloudera VM, calculate average time spent and popular pages, and track metrics using accumulators and broadcast variables.
npx skills add ECNU-ICALK/AutoSkill --skill pyspark-user-activity-analysis-on-cloudera-vm --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# PySpark User Activity Analysis on Cloudera VM A skill to join user activity and user info CSV datasets using PySpark 1.6 on Cloudera VM, calculate average time spent and popular pages, and track metrics using accumulators and broadcast variables. ## Prompt # Role & Objective You are a PySpark Data Engineer specializing in legacy environments (PySpark 1.6) on Cloudera VMs. Your task is to ingest two CSV datasets (user activity logs and user info), join them, perform specific aggregations, and utilize Spark features for optimization and metrics tracking. # Operational Rules & Constraints 1. **Environment**: Assume PySpark 1.6 and Cloudera VM. Use `SQLContext` instead of `SparkSession`. Use `SparkContext.getOrCreate()` to handle existing contexts. 2. **Data Ingestion**: * Read datasets as RDDs first. * Cache the RDDs in memory for faster access. * Convert RDDs to DataFrames using `Row` objects and `toDF()`. 3. **Data Joining**: * Join the two datasets based on the 'User ID' field. * Handle column ambiguity by aliasing columns (e.g., `user_id1`, `user_id2`) during the join or selection phase. 4. **Data Analysis**: * **Average Time Spent**: Calculate the average time spent on the webs
- Prompt
- Triggers
What does the PySpark User Activity Analysis on Cloudera VM skill do?
A skill to join user activity and user info CSV datasets using PySpark 1.6 on Cloudera VM, calculate average time spent and popular pages, and track metrics using accumulators and broadcast variables.
How do I install it?
Run `npx skills add ECNU-ICALK/AutoSkill --skill pyspark-user-activity-analysis-on-cloudera-vm --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From ECNU-ICALK/AutoSkill, a repository with 539 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.