Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
187 changes: 187 additions & 0 deletions .github/skills/migrate-sdk-typespec/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,187 @@
---
name: migrate-sdk-typespec
description: '**WORKFLOW SKILL** - Update SDK project and TypeSpec project, regenerate SDK, to fix error on SDK project. USE FOR: "migrate sdk lib for <tsp-project>".'
---

# Skill Instructions

## Request

The request would come in the form of:
- "Migrate sdk lib for <tsp-project>"

The "<tsp-project>" would be a relative path to a folder in specs repo. A typical form of it can be "specification/web/resource-manager/Microsoft.Web/AppService".

## Checklist

- Files in "autorest.java" repo and folder are not relevant to this skill. Do not read or modify any files in "autorest.java" repo and folder, except this skill folder.
- For TypeSpec project, only update "client.tsp" file and "tspconfig.yaml" file.
- When updating "client.tsp", always use "java" scope, and add new lines to the end of the file.
- For SDK project, do not modify Java file that contains `// Code generated` in the header comment. These are Java files generated by the emitter from TypeSpec project, and should not be modified by agent individually.
- Run `tsp format <tsp_file>` command, whenever you modify the .tsp file, to make sure the format is correct.

## Required repositories

- [Azure/azure-sdk-for-java](https://github.com/Azure/azure-sdk-for-java), cloned at "../azure-sdk-for-java"
Call this folder "sdk repo" for short.
- [Azure/azure-rest-api-specs](https://github.com/Azure/azure-rest-api-specs), cloned at "../azure-rest-api-specs"
Call this folder "specs repo" for short.

You have full access to these locally cloned repositories/folders.

## Steps

### Before you start

<!-- Make sure both sdk repo and specs repo are on a "migrate-<tsp-project>" branch. If not, checkout latest "main", and make a new branch from it. -->

Run `npm ci` in "eng/common/tsp-client" folder in sdk repo to make sure the tsp-client is installed.

Run `npm ci` in specs repo to make sure all dependencies are installed.

### Find the TypeSpec project in specs repo

Find a "tspconfig.yaml" file under "<tsp-project>" folder in specs repo. This file indicates the folder is a TypeSpec project.

"<tsp-path>" refers to the full path of this TypeSpec project folder in specs repo.

### Correct "tspconfig.yaml"

Only modify the YAML under "options"/"@azure-tools/typespec-java" property.

- Remove ".generated" segment from "emitter-output-dir" and "namespace" properties.
- Make sure these options are set
```yaml
premium: true
client-side-validations: true
enable-sync-stack: false
```
- If `unknown` appears in .tsp files, add option `use-object-for-unknown: true`
- If `uuid` appears in .tsp files, add option `uuid-as-string: false`
- Read "readme.java.md" if available, add options of `rename-model`, `add-inner`, `remove-inner`, `preserve-model`, `resource-collection-associations` into "tspconfig.yaml". See [options.ts](https://raw.githubusercontent.com/Azure/autorest.java/refs/heads/main/typespec-extension/src/options.ts) for details of these options.

### Find the project in sdk repo

Now you have the correct "tspconfig.yaml" file, you can find the corresponding SDK project in sdk repo.

Read "emitter-output-dir" property and "service-dir" property (use "options"/"@azure-tools/typespec-java"/"service-dir" if available, otherwise use "parameters"/"service-dir"), combine them for "<sdk-project>". The typical form of it can be "sdk/appservice/azure-resourcemanager-appservice".

"<sdk-path>" refers to the full path of the "<sdk-project>" folder.

"<sdk-service>" refers to the middle segment of the "emitter-output-dir" property in "tspconfig.yaml" file. The typical form of it can be "appservice".

"<sdk-package>" refers to the last segment of the "emitter-output-dir" property in "tspconfig.yaml" file. The typical form of it can be "azure.resourcemanager.appservice".

### Initial mitigation

Here is a list of commonly encountered names used in .tsp that need to be mitigated via `@@clientName(<model-or-property-or-method>, "<correct-name>", "java");`:

```
eTag: etag
userName: username
metaData: metadata
timeStamp: timestamp
hostName: hostname
webHook: webhook
coolDown: cooldown
ID: id
IP: ip
ARM: arm
API: api
```

Use PascalCase for model names. E.g. `ResponseMetaData` to `ResponseMetadata`.
Use camelCase for property and method names. E.g. `acrUserManagedIdentityID` to `acrUserManagedIdentityId`.

### Iterate to fix compile errors

#### Generate the Java code

Run
```sh
python eng/automation/generate.py -s <sdk-service> -c <tsp-path>/tspconfig.yaml
```
in SDK repo root folder.

This script will generate the Java code, and build the Java lib. It may encounter compile error that you need to fix in next step.

PS: this script can take a while to run. If you wait for the command, wait a maximum of 20 minutes.

#### Fix compile error

See [Solve Compile Error](./solve-compile-error.md) for common compile errors and mitigation.

If you work on it for a while, but does not make progress, pause and summarize the errors.

### Modify "module-info.java"

Add this line to "module-info.java" file.
```java
opens com.azure.resourcemanager.<sdk-service>.implementation.models to com.azure.core;
```

### Iterate to fix breaking changes

#### Build the Java lib

Run
```sh
mvn install -pl com.azure.resourcemanager:<sdk-package> -am -DskipTests
```
in SDK repo root folder.

It may encounter error from revapi check. This means there are breaking changes compared with the previous version, and you need to either suppress, or fix in next step.

#### Suppress or fix revapi error

See [Solve Revapi Error](./solve-revapi-error.md) for mitigation.

When you finished this step, pause and output report on remaining errors.
Use the `git diff` with "main" branch, to provide the details on what you think be the cause of the error.

### Update "CHANGELOG.md"

This step should be invoked by user, not by agent.

Remove all generated content in the section of latest lib version of "CHANGELOG.md" file. Also, update the date to "Unreleased".

Then, add a new "Breaking Changes" sub section in this lib version section. The general items of this would be supplied by user.

```
### Breaking Changes

- Removed `<class>` class. <reason>
- Removed `<method>` method from `<class>` class. <reason>
- Changed `<method>` method to `<new_method>` in class `<class>` class. <reason>
etc.
```

### Create pull request and finalize the sdk lib

This step should be invoked by user, not by agent.

#### Verify filename is consistent on GitHub and on filesystem

Make sure the filename on GitHub is consistent with the filename on filesystem (typical reason is that Windows filename is case-insensitive, but git is case-sensitive). If not, use `git mv` command to rename the file on git to make them consistent.

#### Revert change to "eng/versioning/version_client.txt" file and "sdk/resourcemanager" folder

Checkout the content from "main" branch. We are not releasing the lib, hence no need to update the versions.

Use `git checkout main --no-overlay -- sdk/resourcemanager/azure-resourcemanager` to revert "sdk/resourcemanager" folder.

#### Create pull request on specs repo

Create a draft pull request in specs repo.

#### Update "tsp-location.yaml" file, create pull request on sdk repo

When the pull request on specs repo is ready, get the SHA of the last commit.

Update the "commit" property of the "tsp-location.yaml" file in sdk repo, to make it point to this SHA.

Commit the change, and create a draft pull request in sdk repo.

#### Update "tsp-location.yaml" file the last time

When the pull request on specs repo is merged, update the "commit" property of the "tsp-location.yaml" file in sdk repo to point to the SHA on latest "main" branch of specs repo.
16 changes: 16 additions & 0 deletions .github/skills/migrate-sdk-typespec/solve-compile-error.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Solve Compile Error

## "cannot find symbol" or "cannot be converted to"

There are a few reasons, and the mitigation:

- The case difference between .tsp model/property/method and Java code. Use `@@clientName` to mitigate it.
- The model is read-only in .tsp, but the Java code calls the setter method. Please double check whether the Java class indeed contains the corresponding getter method (e.g. `ipRules` for `withIpRules`). This can be mitigated by `@@usage(<model>, Azure.ClientGenerator.Core.Usage.input, "java");`.

## "type argument is not within bounds of type-variable InnerT"

You would need to customize the class.

1. Add option "customization-class: customization/src/main/java/<Service>Customization.java" to "tspconfig.yaml".
2. Create the "customization" module, write the "<Service>Customization.java" class in "sdk-path". Use [this](https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/keyvault/azure-resourcemanager-keyvault/customization) as reference.
3. Commit the changes to "customization" module. Whenever you make changes to the "customization" module, you need to commit it before you run another `python eng/automation/generate.py` command.
103 changes: 103 additions & 0 deletions .github/skills/migrate-sdk-typespec/solve-revapi-error.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
# Solve Revapi Error

The revapi configuration file is "eng/lintingconfigs/revapi/track2/revapi.json" in sdk repo.

The goal in this process is not to suppress every revapi error. It is to mitigate the breaks from "client.tsp" in specs repo.
It is OK that you cannot solve all of them and give a summary.

## Error that can be suppressed

DO NOT suppress revapi, if it does not fall in these categories.

### "java.method.visibilityReduced" and "java.method.removed" on `<init>()` or setter (`with###` method in model class)

```json
{
"code": "java.method.visibilityReduced",
"old" : {
"matcher": "regex",
"match": "method void com\\.azure\\.resourcemanager\\.<sdk-service>\\.models\\..*\\:\\:\\<init\\>\\(\\)"
},
"justification": "Output-only immutable models' constructors are now private."
}
```

```json
{
"code": "java.method.visibilityReduced",
"old" : {
"matcher": "regex",
"match": "method .* com\\.azure\\.resourcemanager\\.<sdk-service>\\.models\\..*\\:\\:with.*\\(.*\\).*"
},
"justification": "Output-only immutable models' setters are now package-private if it's being used by child class."
}
```

```json
{
"code": "java.method.removed",
"old" : {
"matcher": "regex",
"match": "method .* com\\.azure\\.resourcemanager\\.<sdk-service>\\.models\\..*\\:\\:with.*\\(.*\\).*"
},
"justification": "Output-only immutable models' setters are removed if no explicit usage."
}
```

### "java.class.removed" on List/ListResult/Collection class

```json
{
"code": "java.class.removed",
"old" : {
"matcher": "regex",
"match": "class com\\.azure\\.resourcemanager\\.<sdk-service>\\.models\\..*ListResult"
},
"justification": "Pageable models moved to implementation package. Unlikely used by user."
}
```

### "java.missing.oldSuperType" and "java.missing.newSuperType"

Use a narrower regex on class, if possible. E.g. `.*Resource`.

```json
{
"regex": true,
"code": "java\\.missing\\.(oldSuperType|newSuperType)",
"old" : "class com\\.azure\\.resourcemanager\\.<sdk-service>\\.models\\..*",
"justification": "TypeSpec fix for base resource class."
}
```

## Error that are candidates for fix

### "java.class.removed" and "java.method.removed" and "java.field.removed"

Check if this is caused by a naming difference, that can be fixed by `@@clientName(<model-or-property-or-method>, "<correct-name>", "java");` in "client.tsp".

The "CHANGELOG.md" file can be helpful, if e.g. you see
```md
"models.<ModelName>" was removed, and there is a similar "models.<NewModelName>" was added
```
or
```md
"<PropertyName>()" was removed, and there is a similar "<NewPropertyName>()" was added, in same "<ModelName>"
```
This indicates a rename is necessary.

Particularly, if you see
```md
"java.lang.Float" -> "java.lang.Double"
```
in "CHANGELOG.md", add "float32-as-double: false" to "tspconfig.yaml".

A `git diff` with "main" branch can also help you see what's changed in a class.

Whenever you've fixed in "client.tsp" or "tspconfig.yaml", run "Generate the Java code" step to update Java code (so that "Build the Java lib" would now build on the updated Java code).

## Error that should be reported

### "java.class.nowFinal" and "java.class.noLongerInheritsFromClass"

This should be reported.
Loading